TODO

General TODO
============

Build
-----
- Attempt to set up an 8-bit STM8 build (specifically, targetting the
  STM8S-DISCOVERY - UM0817). This will involve making some code changes
  to make sdcc happy (which is a good thing in-and-of itself). It may
  also be worth considering setting up some kind of SDCC test target.

Repo Maintenance
----------------
- Add license text to all source files.
- Consider updating project to use GPL or Apache license.
- Ensure that license is copied into all deliverables (for instance,
  the amalgam and Arduino sketch build directories/archives as well
  as chip8.fth emulator example, if it is copied).
- Consider whether the Arduino and chip8.fth examples should have the
  same license as other code. It may make sense for these examples to
  be permissively licensed (ie. under the MIT license) even if the
  rest of the repo is not.
- Add C++ include guards to all headers?
- Add logo (possibly color and monochrome) where appropriate.
- Consider adding a HACKING.rst document and/or a style guide?


Version 0.3
===========

Build
-----
- [DONE] Split the e4__num namespace into e4__double and e4__usize.
- [DONE] Set up example REPL build for the Arduino MKR Zero.
- [DONE] Make it possible to build REPL, tests, and build tests without
  libffi. This can be done with some kind of Makefile flags and by
  splitting the libffi ldflags into a separate variable that depends on
  libffi. It may be acceptable to require libffi for the build tests,
  but the unit tests and repl absolutely should not require it.

Compilation
-----------
- [DONE] Add conditional structure words CASE OF ENDCASE ENDOF etc.

Documentation
-------------
- [DONE] Create initial documentation framework with Sphinx.
- [DONE] Create initial documentation license.
- [DONE] Create initial background summary.
- [DONE] Complete standard documentation intro.
- [DONE] Add initial system label. This can be kept up to date over
  time.
- Complete kernel standard documentation.
- Complete CORE standard documentation.
- Complete CORE-EXT standard documentation (as much as possible).
- Document version scheme somewhere, which is essentially that tagged
  releases are tagged and version number is not bumped between releases
  but "-dev" is appended to the version string. Consider switching to
  even patch numbers for releases and odd patch numbers (followed by
  "-dev" in the version string) for development versions.
- State that all e4 related variables must be stored in the same
  address space. This limitation makes it impossible to store constant
  values in flash on architectures where ROM variables must be stored
  in a different address space (for example, some 8-bit AVR chips). In
  theory, it might be possible to work around this to some extent by
  providing hooks to allow the user to define things like e4__DEREF to
  check how a variable should be accessed and e4__ROM to declare a
  constant as belong in ROM (on AVR, this could potentially be done
  using some custom linker script and __flash; it may also be possible
  to somehow use __memx in the type for e4__cell and work around needing
  to set e4__DEREF entirely), but for now support is limited to systems
  that can (somehow) fit all of e4 into one address space (which, for
  now, unfortunately means no ATMega328p etc).
- Note somewhere that all divisions are symmetric by default.
  Alternatively, consider the benefits of switching to floored division
  by default.
- Document why there are no plans to implement the FLOAT word set. The
  reason is that it would be difficult/impossible to do so in portable
  ANSI C. C89 does not require IEEE-754 binary representation for
  floating point values, nor is it possible to completely check at
  compile time whether this representation is used (especially when
  accounting for endianness etc). Not only does the Forth 2012 standard
  require this representation, if it cannot be guaranteed then certain
  standard words (such as REPRESENT) may be difficult/inefficient to
  implement using ANSI C standard library floating point utility
  functions alone. Furthermore, even assuming these ANSI C standard
  library floating point utility functions were sufficient,
  unfortunately they are not required by POSIX to be async-signal safe,
  meaning any FLOAT wordset implementation making use of them would not
  be compatible with the current repl implementation.
  Short of implementing everything from scratch "soft float" style,
  which would neither be efficient nor trivial, it would be very
  difficult to support floating point words in a portable way. In
  theory, a portable soft float library could be written based around
  the following structs:
        struct e4__ieee754_float {
            char data[4];
        };
        struct e4__ieee754_double {
            char data[8];
        };
  This may be worth doing at some point, but it is far from a priority
  currently so to say that it's on the back burner would be overly
  generous; for the time being, the pot needed to make the soup is still
  hanging over the sink.
- Note that when calling e4__evaluate, if an ABORT" exception is thrown,
  with message, it is the callers responsibility to display the message
  if they want to do so. The message is available via the
  e4__task_last_abortq API. If exceptions are disabled completely or if
  the top exception handler is the handler in e4__evaluate_quit (ie.
  e4 is being run interactively), an attempt will be made to write the
  message using type IO handler, if available. Much like abort or any
  other exception, it is also the responsibility of the caller to clear
  the stack or cancel compilation as is appropriate.

General Builtins
----------------
- [DONE] Implement S>D.
- [DONE] Implement FM/MOD and SM/REM.
- [DONE] Implement M* and UM*.
- [DONE] */ and */MOD.
- [DONE] Implement <#, #>, #, #S, HOLD, HOLDS, and SIGN.
- [DONE] Fix 2* and 2/ implementations. These words are meant to perform
  arithmetic shifts by one, not simply logical shifts by one.
- [DONE] Implement the PAGE builtin using escape sequences.
- [DONE] Implement the AT-XY builtin using pictured numeric output and
  escape sequences.
- [DONE] Implement the KEY? builtin.
- [DONE] Implement TIME&DATE.
- [DONE] Implement MS.
- [DONE] Move the C" builtin to CORE-EXT, where it belongs.
- [DONE] Implement [IF], [ELSE], and [THEN].
- [DONE] Implement ABORT".
- [DONE] Implement S" and S\" interpretation semantics.
- [DONE] Fix broken DOES> implementation.
- [DONE] Add DOES> tests that cover the actual semantics correctly and
  prevent an incorrect implementation (as was present before).
- [DONE] Fix broken +LOOP implementation.
- [DONE] Add +LOOP tests that cover the actual semantics correctly and
  prevent an incorrect implementation (as was present before).
- [DONE] Uses system LOOP_INC builtin in +LOOP builtin impl.
- [DONE] Implement ENVIRONMENT?.
- [DONE] Implement STATE and >IN.
- [DONE] Implement SOURCE.
- [DONE] Implement UM/MOD.
- [DONE] Implement ACCEPT.
- [DONE] Implement FIND.
- [DONE] Verify that all CORE builtins have been implemented.
- [DONE] Add DLSYM builtins and associated kernel/IO APIs.
- [DONE] Add libffi FFI builtins and associated kernel APIs.
- Consider making it possible to pass callbacks to C libraries using the
  libffi closure API. This would definitely lead to threading concerns,
  however, if e4 code were going to execute on multiple threads.

General IO
----------
- [DONE] Add initial PNO utility functions.
- [DONE] Implement pictured numeric output kernel APIs.

Kernel
------
- [DONE] Add support for multiplying into a double cell number.
- [DONE] Add support for dividing a double cell number by a single cell
  number and obtaining a single cell result (needed for FM/MOD and
  SM/REM).
- [DONE] Add support for dividing a double cell number by a single cell
  number and obtaining a double cell result (needed for #).
- [DONE] Combine sdiv and smod utility functions into one function.
- [DONE] Improve exception printing implementation and other hackish
  printing implementations (such as the . builtin etc.) by using
  pictured numeric output instead.
- [DONE] Move MS and TIME&DATE handlers into e4/core behind a POSIX
  hooks inclusion flag and update build tests accordingly.
- [DONE] Improve stack size calculations.
- [DONE] Handle stack overflow.
- [DONE] Handle return stack overflow.
- Make e4__mem_number behave reasonably when it is passed a string
  with a number that is excessively large.
- Use checked allocation everywhere that HERE is manipulated. WORD is
  specifically allowed to dip into the PNO buffer but not PAD.
  Compilation also does not currently fail correctly/gracefully when
  there is no remaining space.
- Ensure, somehow, that the PNO buffer is always at least 33 characters
  (which is a requirement of WORD). On a system where e4__USIZE_BIT is
  greater than or equal to 16, this will always be the case.
- Confirm that all overflows are now handled.
- Once PNO buffer always works correctly, create file exception
  formatting utility using PNO that outputs information similar to
  the current hack in the repl, then use that instead of the hack
  in both the repl and when implementing INCLUDED etc.
- Remove non-pno numeric formatting utilities.

REPL
----
- [DONE] Make it so that it is possible to automatically include/execute
  a file that has been specified as a command line argument. That is
  to say, doing this:
        $ e4 foo.fth
  should produce the same result as doing this:
        $ e4
        s" foo.fth" included
  (once the later has also been implemented).
- [DONE] Create REPL task buffer size to something much larger (32K?
  8M?) and/or allow for setting buffer size via a command line argument
  or environment variable.
- Look into SIGINT return to shell after CTRL-C followed by graceful
  exit. This may be happening as a result of the longjump out of the
  signal handler, although it *doesn't* seem to happen for signals
  other than SIGINT. Simple work-arounds (like setting SIGINT's action
  to SIG_IGN and then raising SIGINT before exiting the program) do
  not seem to resolve the issue.
- Source ~/.e4-editrc to allow for e4 specific input overrides. For
  example, tab completion on tab could be disabled by macro binding
  the tab key to two or four spaces.
- [DONE] Add ARGV and ARGC words. ARGV should return a string and its
  length, similar to s", if the argument is valid. ARGC should just be
  argument count.

Repo Maintenance
----------------
- Update README.rst to actually explain this project.
- Consider renaming wip branches to dev.

Testing
-------
- [DONE] Split e4t__test_util_doublemath into multiple tests after the
  num to usize/double split.
- Consider adding case sensitive e4t__ASSERT_MATCH variant and updating
  tests to use it where appropriate.


Version 0.4
===========

Build
-----
- Update amalgam build target to be a .zip file (rather than e4.c) and
  produce this file when building? Alternatively, create some kind of
  "dist" make target that creates a zip file including the amalgam as
  well as full license text and PDF/HTML/RST documentation.

General Builtins
----------------
- Implement remaining CORE EXT builtins.
- Add FILE and some FILE EXT word sets impls. Ensure that behavior
  of relevant words (such as SOURCE-ID, REFILL, and () is also updated
  as appropriate).
- Fix INCLUDED so that it actually scans a reasonable search path (ie.
  the current working directory of either the file being evaluated or
  the program itself). This will likely require new IO handlers. The
  easiest way to implement this might be to add handlers to get the
  working directory of an open file and handlers to get and set the
  current working directory. Unfortunately, this will require storing
  the directory of a file path when it is opened, which will require
  even more messing with handlers. It also wouldn't work with
  e4__evaluate_file, which is a shame. At the end of the day, there
  just isn't a great way to do this that doesn't involve having to
  manually track each open file and maintain a mapping. The alternative
  is to somehow make things work *only* with _evaluate_path, which
  could *probably* be forced to work by storing path in the IO source
  (although the lifetime of path is not long enough). The best thing
  would probably be having some _io_file_dir_from_path handler that
  could be called on paths in e4__evaluate_path. This could then
  update the io_src for input with a directory descriptor. If this
  descriptor is set on subsequent calls to evaluate, some sort of
  openat and/or chdir can be called. The easiest thing would probably
  be to make the current open function some kind of openat function
  and just allow passing it some kind of RELATIVE_OPEN flag. We can
  then maintain our own DIR_IS_VALID flag in io_src and only set it
  after a successful call to _io_file_dir_from_path. We will probably
  also need to add the option of a separate function to call to close
  the directory handler, but for our posix purposes we can just
  reuse our existing handler. For POSIX purposes, we can use dirname.
  So, essentially, we have this:
     * open becomes some form of openat that takes an optional
       directory handler. This really only requires a flag; we already
       have fd as an out parameter, so we can just look here if we are
       instructed to do so and call openat internally.
     * Add an _io_file_openparentdir function that returns a directory
       handler. We'll call this in evaluate_path and store it in the
       io_src. In evaluate path, if flags in io_src indicate, then
       we'll use it in openat. This needs to *also* take an openat
       style directory descriptor to account for the fact that we may
       already be in openat territory.
     * Add an handler for closing directory handles.
  A little bit of a mess, but it should be doable.
- Consider adding _NF variants to USERVAR declaration macros, defining
  e4__UV_BEGIN, and simply defining BEGIN as its own uservar rather
  than in terms of HERE using PUN macros. Consider also removing pun
  macros.
- Consider adding a LOCALS wordlist implementation. This can be done by
  taking the following measures:
    * When compiling, maintain a mapping of locals to offsets. We can
      limit the names of locals to 31 characters each per 3.3.1.2.
      These locals should run as some immediate which compiles into
      LOCAL_LIT,i (or something along those lines) that will access
      local rbp[i] at runtime.
    * On local definition, >r is compiled.
    * At the end of compilation, r> (or some return stack popping
      variant) are compiled in for each locals dictionary entry.
    * TO looks ahead when compiling already. If the word it looks
      ahead to is a local, it should compile as LOCAL_STORE,i (or
      something along those lines).
    * This should make it possible to make definitions like as follows:
        : foo [ s" my-local" ] (local) ;
    * 0 0 (local) can probably be a no-op with this implementation.
    * Storage for local mappings during compilation can be stored in
      some specially reserved block in the task structure. 2K should
      be sufficient for 16 entries. It may be possible to get away
      with using less space. State tracking such as number of locals
      defined and true dictionary top (without compile-time locals
      definitions)can live here as well. mem_dict APIs can probably be
      used.
    * A #LOCALS environmental query should be added.
  Once these measures have been taking and (LOCAL) is functional, words
  like LOCALS| and {: can be added as well.

Documentation
-------------
- Note that all file IO handlers must set a non-zero platform specific
  error code in *ior (to later be read with e4__task_ior) when returning
  e4__E_FILEIO or any other return code for which a platform IO error is
  expected (see e4__E_HAS_PLATFORM_IOR in e4.h).

Kernel
------
- [DONE] Consider providing optional POSIX handlers for file IO. These
  could be enabled/disabled at compile time.
- Consider gating e4__io_dump and e4__io_see behind e4__INCLUDE_TOOLS.

REPL
----
- Try running some the Tetris program again.

Testing
-------
- Consider splitting exceedingly long builtin tests into multiple
  functions. Current candidates for splitting include doublemath,
  logic, math, and memmanip in particular.
- Consider splitting builtin tests into multiple files, possibly even
  in a builtin/ test directory.
- Consider adding mock file IO tests that do not rely upon the POSIX
  handlers.
- Consider splitting exceedingly long builtin tests into multiple
  functions. Current candidates for splitting include doublemath,
  logic, math, and memmanip in particular.
- Consider splitting builtin tests into multiple files, possibly even
  in a builtin/ test directory.


Version 0.5
===========

Compilation
-----------
- [DONE] Implement decompilation (ie. SEE). To start, simply naively
  display the internal representation. The fully naive approach, for
  words with the threaded_execute handler, is to simply search the
  dictionary linearly for each word to see if any entry has each address
  as an execution token. If it does, display the name corresponding to
  that entry. If not, simply display the address. This isn't terribly
  efficient, but it doesn't require storing any references from XT to
  dictionary entry or any other metadata, so it is probably sufficient.
  Output can look something like this for threaded code.
    : foo 2 2 dup . ; see foo
        : foo ( threaded - user: 0x0 )
            LIT_CELL
            0x2
            LIT_CELL
            0x2
            DUP
            .
            EXIT
            SENTINEL

Kernel
------
- Add full support for double cell numbers (including arithmetic, the
  text interpreter extension for numbers of the format XXXX. as outlined
  in section 8.3.1 of the standard, etc.) by implementing all (or most)
  of the DOUBLE word set.
- Consider whether it might be possible to wrap the main quit loop in
  some kind of exception context (somehow?) to get around the race
  condition created by using multiple different exception contexts
  within that loop. It's unclear exactly how this would look since
  the quit loop relies on certain exceptions percolating all the way
  up, so for it to itself run in an exception context some considerable
  refactoring might be necessary. This is important because on embedded
  systems, or even in a repl where a signal can be caught, it may be
  desirable to throw an exception when quit, as currently implemented,
  is not in an interrupt context. This *may* not be an issue (failure
  to actually throw is always something an API client can work around),
  but it probably isn't entirely desirable.
- Add support for adding a single cell number to a double cell number
  (which is needed for >NUMBER). It may be assumed that the single cell
  number is signed (as in M+).
- Update words meant to support double width numbers that are currently
  implemented without any such support. This includes >NUMBER.


Version 0.9
===========

(things to have done prior to 1.0)

Build
-----
- Split system required words and optional system specific extension
  words into different modules (SYSTEM and SYSTEM-EXT?).

Compilation
-----------
- Implement AHEAD?
- Implement some system +LOOP word that executes LOOP/LOOP+ execution
  semantics to save RAM space? This can be done using the same
  return stack manipulation techniques used for words like R@ and J.
- Ensure that ] is standards compliant. It seems that it should be, but
  there is a somewhat confusing note in second 6.1.2250 "STATE" that
  may require more investigation.

Documentation
-------------
- Review/fix license.
- Complete "about" documentation.
- Complete project overview documentation.
- Complete REPL documentation.
- Complete quickstart documentation.
- Complete testing documentation.
- Complete API reference documentation.
- Complete Standard documentation.
- Consider adding README.rst files and release notes to documentation
  (somehow?).
- Consider including genindex in some capacity (if this would be
  helpful).
- Document all implementation-defined behavior as per Forth 2012 4.1.1.
- Document all ambiguous condition handling as per Forth 2012 4.1.2.
- Provide other system documentation as per by Forth 2012 4.1.3.
- Add project logo to documentation.
- Document the fact that excluding the exception wordlist does not
  actually disable exceptions. Even with the exceptions wordlist
  excluded, exceptions are still used internally; it's just that the
  CATCH and THROW builtin words are not available.

Kernel
------
- Consider adding optional IO handlers to allow for supporting the
  FILE wordset. If this is done, also add FILE related builtins
  and the appropriate hooks to the REPL.
- Consider adding builtin POSIX handlers for file IO that can be
  enabled/disabled (off by default) at compile time.
- Consider making return stack coherence tracking optional at compile-
  time to allow for RAM savings.
- Throw address alignment exceptions (-23) where appropriate? On one
  hand, this could be a nice way to avoid performance issues. On the
  other hand, on some systems unaligned access is fine.
- Consider adding an e4__stack_rclear API and using where appropriate.
- Consider adding some kind of NO_INTERPRET or INTERNAL flag for words
  that can never be used correctly (whether interpreting or compiling)
  from a non-system context (for instance, BRANCH, BRANCH0, and LIT_*
  as currently implemented). Making words that need to have some effect
  compile only and immediate then checking whether compiling as a hack
  is not sufficient because those words may be called from a word
  marked IMMEDIATE. Alternatively, some other word to bake in addresses
  (such as [LIT]), which risky, could be handy. With such a word, it
  would be possible to write things like : SKIP BRANCH [LIT] 1 ; (which
  itself is also more risky than useful, although that doesn't mean
  there could be no reasonable application). As an alternative, these
  words could be given some kind of "hidden" flag so that it isn't
  possible to look them up.
- Consider either enhancing the ability to run e4 without exceptions
  enabled or removing that ability completely. That is, consider making
  it undefined behavior to call the e4__execute family of functions and
  all builtins without exceptions enabled (and removing all associated
  handling that currently exists), or adding more facilities to make it
  possible to disable exceptions completely at a global level. This
  would likely involve adding some mechanism for reporting errors
  (perhaps some sort of last_error field in the e4__task struct) in
  cases where failure is currently silent. Alternatively, leaving
  support partial/tentative indefinitely is also a reasonable option.

General Builtins
----------------
- Consider implementing the entire Programming-Tools word set.
- Determine whether CR should actually output \r\n. If it should,
  ensure that \r\n is used instead of \n in all places where \n is
  currently in use.

REPL
----
- Split REPL source into multiple files? (For instance, one file for
  auto-completion related logic, one file for signal handling related
  logic/init/uninit/etc., one file for e4 IO handlers, main.c, and
  a header file to tie it all together).

Testing
-------
- Add tests for every builtin word as prescribed by the standard.


Transfer Station
================

(won't do lists; cleared after each release)

Build
-----
- Attempt to set up an AVR build (or some other 8-16 bit target build)
  to determine ROM size in that environment. (Preliminary size optimized
  builds seem to be at around 21K when targeting the ATMega328p).
    -- Unfortunately, because it is not trivial to store constants in
       flash memory and read them back in the same way as variables
       stored in SRAM on the 8-bit AVR archetecture, this is not
       trivial. Punting for now in favor of an STM8 build.

Kernel
------
- Add more two's complement utility macros as needed and use them where
  appropriate. -- Wow, this really needs to go away. We have plenty of
  two's complement utility macros.
- Consider the benefits of making the word parsing/PNO area a fixed
  buffer (similar to the area returned by PAD) rather than simply using
  the memory pointed to by HERE. Currently, word parsing can overflow
  the dictionary and there may not be enough space available to provide
  a buffer as large as the standard requires. On the flip side, the
  benefit to the current approach is that it is possible to use the
  memory that would be reserved for the PNO buffer for something other
  that pictured numeric output (ie. additional definitions) if that is
  for some reason desired on a severely memory constrained system.
  Presumably the same argument could be made about the dedicated region
  returned by PAD (currently implemented as a fixed region), which is
  also something to consider.
    -- The PNO buffer need not be fixed, but the minimum required space
       should always be reserved. There is no harm in a fixed PAD
       because while there is no standards compliant way to check
       how much space is available in PAD, that amount of space may not
       be variable.
- Consider actually typedefing e4__double. Normally structs in e4 are
  not typedefed, although since struct e4__double is relatively small
  and is always passed/returned by value rather than by reference, it
  may make sense to simply typedef away the fact that it is a struct
  in the first place as a special case. I'm currently leaning towards
  not doing this, but it is still worth consideration.
    -- This would just be too inconsistent with the way that structs
       are treated elsewhere in e4 and would violate the principal of
       least astonishment for no real reason. Better to just not do it.

Repl
----
- Try running the modified ANS 94 Forth Tetris program from Rosetta
  Code.
    -- This doesn't work well yet. Perhaps it will be worth revisiting
       once the remaining words from CORE, CORE EXT, and the file word
       set have been implemented.
- Re-evaluate not printing a newline on enter (and just simply printing
  a space or something). This would look far cleaner.
    -- No. This isn't going to happen without completely abandoning
       libedit and making some kind of bespoke line editing system,
       which absolutely isn't worth it, because that isn't the goal of
       this project. It works fine as it stands; this action item is
       out of here.