Create compiler user guide

compiler-user-guide/docs/basic-usage.md

@@ -0,0 +1,35 @@
+# Basic Usage
+
+Theoretically (although there are some [restrictions](restrictions.md)), every defined function in a workspace can have a compiled bytecode form. This bytecode is saved and loaded as part of the workspace, and will be copied along with the function on [`⎕CY`](../../language-reference-guide/system-functions/cy) or [`)COPY`](../../language-reference-guide/system-commands/copy) or the [`⎕OR`](../../language-reference-guide/system-functions/or) of a compiled function.
+
+To query whether a function `foo` has been successfully compiled, enter:
+```apl
+      1(400⌶)'foo'
+```
+
+This returns a Boolean value of 1 if the compilation has been performed.
+
+To compile a function foo, enter:
+```apl
+      2(400⌶)'foo'
+```
+
+This returns a matrix of diagnostic information. If the matrix has zero rows then the function was compiled successfully. Otherwise, each row of the matrix describes a problem that prevented the compiler from compiling the function.
+
+When a function is executed, Dyalog automatically executes any compiled code for the function. If none is available, then the function is executed using the traditional APL parser.
+
+In summary:
+```apl
+      ⎕FX'r←foo y' ... ⍝ define foo
+```
+```apl
+      foo 99           ⍝ execute the uncompiled code
+```
+```apl
+      2(400⌶)'foo'     ⍝ compile it
+```
+```apl
+      foo 99           ⍝ execute the compiled code
+```
+
+For full details on the syntax of `400⌶`, see [Compiler Control I-beam](../../language-reference-guide/primitive-operators/i-beam/compiler-control).

compiler-user-guide/docs/compiling-operators.md

@@ -0,0 +1,30 @@
+# Compiling Operators
+
+When compiling a defined function or operator, the compiler needs to know the nameclass of every name that is used. This presents a problem for defined operators, because the nameclass of the operands is not known:
+```apl
+op←{
+    ⍺⍺ / ⍵ 	  ⍝ ⍺⍺ could be an array or a function
+}
+```
+
+When an operator is compiled using [`2(400⌶)Y`](../language-reference-guide/primitive-operators/i-beam/compiler-control.md#x-2-compile), the compiler assumes that the operands are functions. If the compiled operator is subsequently called with an array operand, then the compiled version is not used and the interpreter uses the parser instead.
+
+To work around this, the compiler can be run in a mode where it will attempt to compile a defined operator the first time it is applied to some arguments; at this point the compiler can see exactly what the operands are. If the compilation is successful, then the compiler will record the nameclass of the operands along with the compiled bytecode. When the operator is applied again, the compiled bytecode will only be executed if the operands still have the same nameclass as they did the first time the operator was applied (if the nameclass of an operand has changed, then the compiled bytecode will not be used and the operator will be interpreted).
+
+Continuing the example:
+```apl
+      400⌶2       ⍝ enable auto compilation of operators
+```
+```apl
+      +op 1 2 3 4 ⍝ op is compiled assuming fn operand
+10
+```
+```apl
+      400⌶0       ⍝ disable auto compilation
+      ×op 1 2 3 4 ⍝ execute compiled bytecode again
+24
+```
+```apl
+      1 2 3 4 op 1 2 3 4 ⍝ operand is array; revert to interpreter
+1 2 2 3 3 3 4 4 4 4
+```

compiler-user-guide/docs/compiling-with-global-names.md

@@ -0,0 +1,26 @@
+# Compiling With Global Names
+
+When compiling a defined function or operator, the compiler needs to know the nameclass of every name that is used. It is useful to distinguish between local names (those that are defined in the function or operator being compiled) and non-local or global names (everything else).
+```apl
+foo←{
+    n←⍵×2            ⍝ define local name n
+    bar n            ⍝ use global name bar and local name n
+}
+```
+
+If this fuction is compiled with [`2(400⌶)` ](../language-reference-guide/primitive-operators/i-beam/compiler-control.md#x-2-compile)`'foo'` , the compiler will determine the nameclass of global names by looking at the names that are currently defined in the workspace:
+
+- If `bar` is undefined, then the compiler will not compile `foo`.
+- If `bar` is defined, then the compiler will use its nameclass to determine whether it is an array, function or operator, and parse the body of `foo` accordingly.
+
+In the latter case, the nameclass of `bar` is recorded in the compiled form of `foo` as a checked assumption; when `foo` is executed, if `bar` no longer exists or has a different nameclass, an error will be reported.
+
+In more complex applications there could be a requirement to restrict the set of global functions and variables that compiled code can refer to, or it might be known in advance exactly which global variables and functions will exist when the application is run. In these situations, the application can be compiled with `N(400⌶)`, where `N` is a namespace containing callback functions that the compiler can use to determine the nameclass of any global names it encounters.
+
+For example, to mimic the behaviour of `2(400⌶)`, the following callback functions can be defined in `#` (the root namespace):
+```apl
+      quadNC←⎕NC ⋄ quadAT←⎕AT     ⍝ define callback fns in #
+      #(400⌶)'foo'                ⍝ pass in # as the namespace
+```
+
+More complicated definitions of the callback functions grant finer control over exactly which global names a compiled function is allowed to refer to.

compiler-user-guide/docs/index.md

@@ -0,0 +1,10 @@
+---
+search:
+  exclude: true
+---
+
+# About
+
+The _Compiler User Guide_ is a complete guide to the compiler, which reduces interpreter overhead, and details the syntax and restrictions of its current implementation.
+
+Assumed knowledge: A reasonable understanding of Dyalog APL.

compiler-user-guide/docs/introduction/changes-to-behaviour.md

@@ -0,0 +1,17 @@
+# Changes to Behaviour of Functions when Compiled
+
+The same run-time engine is used by both compiled functions and interpreted functions when executing primitive functions. However, a small number of behavioural changes occur when functions are compiled.
+
+## Thread Switching
+
+Thread switching will not occur between lines of code after a function has been compiled. However, it can still occur at the start of the function before the first line is executed.
+
+## Error Trapping
+
+Compiled functions cannot be suspended. Errors occurring within compiled functions are signalled back to the calling environment (in the same way as if [`⎕SIGNAL`](../../../language-reference-guide/system-functions/signal) had been used inside the function).
+
+Similarly, when an error in a compiled function is handled by an Execute trap, the APL expression specified in the trap will be executed in the calling environment and will not be able to see any of the compiled function's local names.
+
+## Visible Names
+
+When a user-defined operator is compiled, its local names are eliminated. As a result, the names are no longer visible to any sub‑functions that it calls.

compiler-user-guide/docs/introduction/index.md

@@ -0,0 +1,13 @@
+# Introduction
+
+When the APL interpreter executes a user-defined function, it spends most of its time performing two separate actions:
+
+- Parsing the APL syntax ("interpreter overhead")
+- Executing individual primitive functions
+
+The compiler is designed to reduce the time spent on the first of these two actions – the interpreter overhead – by converting the APL source code into a bytecode form that is more efficient to execute.
+
+The biggest performance gains are achieved when the compiler is used on functions that are applied to simple scalars or small array arguments. If the arguments are large arrays, then the interpreter spends most of its time executing the primitive functions; this means that the benefit of reducing the interpreter overhead is less significant.
+
+!!! Info "Information"
+    Dyalog Ltd is continually researching performance improvement mechanisms; advances in other areas mean that the compiler is unlikely to be developed further as efforts are now being directed elsewhere.

compiler-user-guide/docs/introduction/optimisations.md

@@ -0,0 +1,36 @@
+# Optimisations
+
+In addition to reducing interpreter overhead, the compiler can also perform certain optimisations on the APL code. These include:
+
+- constant folding
+- eliminating local names
+- flexible idiom recognition
+
+## Constant Folding
+
+When a primitive function is applied to constant arguments, the compiler attempts to evaluate the entire expression at compile time, thereby saving time when the function is executed. For example:
+```apl
+encode←{(⎕A,⎕D)⍳⍵}   ⍝ ⎕A,⎕D is evaluated at compile time
+```
+
+However, the compiler cannot always successfully evaluate every expression. For example, the compiler cannot evaluate primitive functions that depend on system variables:
+```apl
+numbers←{⍳7} ⍝ compiler cannot evaluate ⍳7 as it does not
+             ⍝ know what value ⎕IO will have
+```
+
+Constants will only be retained if they are reasonably small. The limit is typically around 1,000 items for a simple array.
+
+## Eliminating Local Names
+
+Every assignment to a local name incurs a measurable overhead, especially within a dfn. The compiler discards all local names as part of its normal operation, so this overhead is eliminated in compiled code.
+
+## Flexible Idiom Recognition
+
+The Dyalog interpreter recognises idioms as specific sequences of characters, for example, `0=⍴⍴x`. The compiler recognises the same idioms but in a more flexible way, enabling it to cope with syntactic variations. This means that expressions can be identified as idioms (and processed as such) even if:
+
+- parts of the expression are named (as long as there are no other uses of the same name), for example, `s←⍴x ⋄ 0=⍴s`
+- redundant parentheses are added, for example, `0=(⍴⍴x)`
+- the arguments to commutative functions are swapped, for example, `(⍴⍴x)=0`
+
+In these situations, the compiler's optimisations transform the expression into one that matches an idiom. For example, `(≢⍬)=⍴⍴x` is recognised as being the same as the idiom `0=⍴⍴x` because the expression `≢⍬` is evaluated to `0` at compile time.

compiler-user-guide/docs/introduction/upgrading-from-previous-dyalog-versions.md

@@ -0,0 +1,3 @@
+# Upgrading from Previous Versions of Dyalog
+
+The format of the bytecode used for compiled functions in Dyalog v15.0 onwards is not compatible with the bytecode used for compiled functions in previous versions of Dyalog. When loading a workspace containing compiled functions that was saved by a previous version of Dyalog, all bytecode for compiled functions will be discarded and you must use [`400⌶`](../../language-reference-guide/primitive-operators/i-beam/compiler-control.md) to recompile them).

compiler-user-guide/docs/restrictions.md

@@ -0,0 +1,97 @@
+# Restrictions
+
+There are several restrictions when using the compiler, some of which might be removed in later versions.
+
+## Restriction 1
+
+__A function that uses semi-global names cannot be compiled.__
+
+To compile a function, the compiler needs to be able to determine the nameclass of every name used in that function so that it knows whether it refers to an array, a function or an operator.
+
+For local names, the compiler can identify the nameclass because it can see the definition of the name:
+```apl
+sum←{
+    f←+         ⍝ compiler sees this definition...
+    f/⍵         ⍝ ... so knows that f is a function here
+}
+```
+
+For global names, [callbacks from `400⌶`](../language-reference-guide/primitive-operators/i-beam/compiler-control.md#x-is-a-namespace-compile-with-callbacks)) enable the compiler to identify the nameclass.
+
+However, for semi-global names (that is, names that are local to the function that calls the function to be compiled) the compiler cannot determine the nameclass:
+```apl
+∇ r←sum y       ⍝ if f is defined in sum's caller, then...
+  r←f/y         ⍝ ...this could be +/y, or 2/y, etc.
+∇
+```
+
+## Restriction 2
+
+__A function that calls system functions which refer to values by name or create new named values cannot be compiled.__
+
+Compiled functions can use local names but, as part of the compilation process, the compiler discards these names, so they do not appear in the compiled bytecode. For this reason, system functions that refer to values by name, or create new named values, are prohibited:
+```apl
+foo←{
+    a←⍺+⍵       ⍝ local name 'a' is discarded by compiler
+    r←⎕NL 2     ⍝ ⎕NL is prohibited as it needs to see 'a'
+    'a'⎕NS''    ⍝ ⎕NS is prohibited as it redefines 'a'
+}
+```
+
+## Restriction 3
+
+__A function that uses the dot syntax for namespace references cannot be compiled.__
+
+The compiler cannot determine the nameclass of a name when the dot syntax is used to refer to names inside arbitrary namespaces:
+```apl
+sum←{
+    ⍺.f / ⍵         ⍝ ⍺.f could be an array or a function
+}
+```
+```apl
+prod←{
+    #.util.prod ⍵   ⍝ nameclass of util and prod unknown
+}
+```
+
+## Restriction 4
+
+__A function that includes certain control structures cannot be compiled.__
+
+The following control structures prevent a function from being compiled:
+
+- [`:Trap`](../../programming-reference-guide/defined-functions-and-operators/traditional-functions-and-operators/control-structures/trap)
+
+- [`:Hold`](../../programming-reference-guide/defined-functions-and-operators/traditional-functions-and-operators/control-structures/hold)
+
+- [`:With`](../../programming-reference-guide/defined-functions-and-operators/traditional-functions-and-operators/control-structures/with)
+
+- [`:Disposable`](../../programming-reference-guide/defined-functions-and-operators/traditional-functions-and-operators/control-structures/disposable)
+
+## Restriction 5
+
+__A function cannot be compiled if it includes certain language features.__
+
+The following language features prevent a function from being compiled:
+
+- dfn error guards
+- localised [`⎕TRAP`](../../language-reference-guide/system-functions/trap)
+- function trains
+
+## Restriction 6
+
+__A function that includes the Execute function (`⍎`) cannot be compiled.__
+
+The compiler prohibits the use of _execute_ ([`⍎`](../../language-reference-guide/primitive-functions/execute)) because it could have arbitrary side effects unknown to the compiler.
+
+## Summary
+
+A function cannot be compiled if it:
+
+- uses semi-global names.
+- calls a system function that refers to values by name or creates new named values.
+- uses the dot syntax between user-defined names.
+- includes the control structures `:Trap`, `:Hold`, `:With` or `:Disposable`
+- includes dfn error guards or localises `⎕TRAP`
+- includes function trains
+- includes the _execute_ function (`⍎`)

compiler-user-guide/mkdocs.yml

@@ -0,0 +1,63 @@
+site_name: Compiler User Guide
+theme:
+  favicon: documentation-assets/images/favicon-32.png
+  logo: documentation-assets/images/dyalog-logo_white.svg
+  features:
+    - navigation.instant
+    - navigation.tracking
+    - navigation.path
+    - navigation.indexes
+  name: material
+  font:
+    text: Be Vietnam Pro
+extra_css:
+  - documentation-assets/css/main.css
+  - documentation-assets/css/extra.css
+extra_javascript:
+  - javascripts/mathjax.js
+  - https://polyfill.io/v3/polyfill.min.js?features=es6
+  - https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
+plugins:
+  - privacy
+  - search
+  - macros
+  - site-urls
+  - caption:
+      table:
+        enable: true
+        start_index: 1
+        increment_index: 1
+        position: bottom
+        reference_text: 'Table {index}'
+        caption_prefix: 'Table {index}:' 
+extra:
+  version_maj: 20
+  version_majmin: 20.0
+markdown_extensions:
+  - admonition
+  - pymdownx.details
+  - pymdownx.keys
+  - pymdownx.superfences
+  - pymdownx.arithmatex:
+      generic: true
+  - pymdownx.highlight:
+      use_pygments: false
+  - attr_list
+  - abbr
+  - footnotes
+  - md_in_html
+  - markdown_tables_extended
+  - toc:
+      title: On this page
+nav:
+  - index.md
+  - Introduction:
+      - introduction/index.md
+      - Optimisations: introduction/optimisations.md
+      - Changes to Behaviour of Functions when Compiled: introduction/changes-to-behaviour.md
+      - Upgrading from Previous Versions of Dyalog: introduction/upgrading-from-previous-dyalog-versions.md
+  - Basic Usage: basic-usage.md
+  - Compiling with Global Names: compiling-with-global-names.md
+  - Compiling Operators: compiling-operators.md
+  - Restrictions: restrictions.md
+