Content Addressable Hashes

[blakeembrey]

I'm making this issue here because things may change and I don't want to notify a lot of people ahead of time with my thoughts.

Goals:

• De-dupe typings
• Manifest files
• Fix module typings conflicts

The idea:

Content addressable hashes. For the next version I'd really like to fix de-duping and re-writing of dependencies which causes issues on import still. I realised one way to solve all of the above is to use content hashes as the file names and just write wrappers to the content hashed .d.ts files. The structure will look more like:

typings/.store/klou3f3u34u982r04jdf.d.ts
typings/main/popsicle.d.ts
typings/main.d.ts
...

The stored content hashed files will be per-typed file and the nature of hashes will avoid duplicated dependencies. No more rewriting typings other than import/export still, and writing a wrapper separately. This is a pretty big architectural change though, so I'll be working on it in core first.

...

declare module 'popsicle' {
  export * from '../.store/klou3f3u34u982r04jdf.d.ts'
}

[unional]

For my own education,

• typings/.store/klou3f3u34u982r04jdf.d.ts is the external module definition
• typings/main/popsicle.d.ts is the wrapper ONLY contains

declare module 'X' {
  export * from '../.store/klou3f3u34u982r04jdf.d.ts'
}

where X is specified in typings.json:

{
  "files": {
     "X": "<path to X definition (e.g. main.d.ts)>"
  }
}

or

{
  "files": [
    "<path to somename.d.ts>"
  ]
}

where X = path.slice(-5); // without '.d.ts'

[blakeembrey]

Where X is the module name given from typings install (so option 1). However, just like currently, there'll be more than one wrapper and there'll be individual file wrappers as well (so also option 1 + option 2 - X/path/to/somename).

[unional]

Ar, of course. It's External Module. :P

The question is we need some anchor point to determine the relative path. e.g. in the final "ambient" version (where core-js is input from typings install:

declare module 'core-js' {
  export * from '<X>';
}

declare module 'core-js/fn/array' {
  export * from '<Y>';
}

The typings.json need to have a baseline that reference to X, and a relative path X/fn/array => Y.
One possible solution is:

{
  files: {
    "main": "X",
    "main/fn/array": "Y",
    "browser": "X",
    "browser/fn/array": "Y"
  }
}

UPDATE: where X, Y are file names defined by typed file creator, and <X> <Y> is content addressable hashes file names of X and Y.

[unional]

Another solution, reusing "main" entry point, is: expanding on the package-browser-field-spec:

{
  "main": {
    "core-js": "X",
    "./fn/array": "Y"
  },
  "browser": {
    // if user should not get the module through main entry point due to impractical file size.
    "core-js": false,
    "./fn/array": "browser-Y"
  }
}

[blakeembrey]

I might be missing something, but what are you trying to solve with this proposal? The relative names will always come from the filename relative to typings.json, you can just put it where you want to match the actual implementation (which is what I actually do currently, if you look at each of the modules I've typed).

Aside from that, I didn't understand the "UPDATE". I don't ever want someone to actually reference the hashed files themselves, it's just an approach to fix some of the issues I'm seeing in the current Typings release.

[unional]

Sorry, may be my English is not doing me much good. 😏

What I tried to describe is a proposal on:

1. How it is defined in the community typed definition project (e.g. typed-core-js)
2. How it will appear in the consuming module. (e.g. my-super-app)
   with regard to multiple export modules.

Let me try it again with an example.
core-js has multiple sub-path exports: core-js, core-js/fn/array, core-js/fn/map etc. They are relative (relate to the base core-js).
In typed-core-js, the file structure is:

+ --- main.d.ts
+ --- main.fn.array.d.ts
+ --- browser.fn.array.d.ts
+ --- typings.json

While normally main.fn.array.d.ts could be organized as main/fn/array.d.ts, we do not need to force that on the community, as long as they can do a proper mapping in typings.json they can name their files whatever they want.

In the typings.json

{
  "main": {
    "core-js": "main.d.ts",
    "./fn/array": "main.fn.array.d.ts"
  },
  "browser": {
    // if user should not get the module through main entry point due to impractical file size.
    "core-js": false,
    "./fn/array": "browser.fn.array.d.ts"
  }
}

In my-super-app, run typings install core-js --name core. Each file are downloaded and renamed based on their content:
main.d.ts -> typings/.store/hashA.d.ts
main.fn.array.d.ts -> typings/.store/hashB.d.ts
browser.fn.array.d.ts -> typings/.store/hashC.d.ts

typings/main/core-js.d.ts:

declare module 'core' {
  export * from '../.store/hashA.d.ts';
}
declare module 'core/fn/array' {
  export * from '../.store/hashB.d.ts';
}

typings/browser/core-js.d.ts:

declare module 'core/fn/array' {
  export * from '../.store/hashC.d.ts';
}

Hope it is better this time. 😏

[blakeembrey]

@unional To be honest, I don't think it's a useful change and will restrict use in the future. Is there a valuable reason you can't replicate the file structure? I'm not convinced it's a problem that needs solving.

Edit: Just image having to maintain this with every feature going forward. Is this an override field? How does it integrate with other filesystem fields? What does main/browser point to? The original or the mapped value? Why does browser map from nice filename to real file instead of nice file to nice file or real file to real file?

[unional]

Are you referring to this?

While normally main.fn.array.d.ts could be organized as main/fn/array.d.ts, we do not need to force that on the community, as long as they can do a proper mapping in typings.json they can name their files whatever they want

I don't see it as override field, unless you mean the browser part. But that is part of the browser-field-spec.

If we solely rely on the folder structure, how would the folder structure and typings.json look like?
I can't visualize that. :(

[blakeembrey]

@unional It's how it already works though? E.g. https://github.com/typings/typed-es6-promise the main typing is in dist, matching the module. In https://github.com/typings/typed-debug, there's multiple entry points that also match the module. I actually type modules pretty closely with the file structure and I think the filesystem is a good place for that information to stay 😉

[blakeembrey]

Ok, I'm going to change this issue to be about content addressable hashes and make a new issue roadmap for 0.7 (or 0.8, etc).

[unional]

Um......still not seeing it. Sorry. 😢

using the core-js example, should the file structure look like this:

+ browser
    + fn
        + array.d.ts
+ main
    + fn
        + array.d.ts
+ main.d.ts

and typings.json:

{
  "main": [
    "main.d.ts",
    "main/fn/array.d.ts"
  ],
  "browser": [
    "browser/fn/array.d.ts"
  ]
}

Or it's something else?

[unional]

In that case how to we describe "in browser the core (main.d.ts) is not available"?
as in:
https://github.com/defunctzombie/package-browser-field-spec#ignore-a-module

[blakeembrey]

It currently works. Using your example above, I'll refactor to the proposed model (which doesn't actually relate to the original discussion anymore).

{
  "main": "main.d.ts",
  "files": [
    "main/fn/array.d.ts"
  ],
  "browser": {
    "main/fn/array.d.ts": "browser/fn/array.d.ts"
  }
}

Main will never be multiple aliases - you can't have two "main" files on import.

[unional]

Do you mean

{
  "main": "main.d.ts",  // <-----
  "files": [
    "main/fn/array.d.ts"
  ],
  "browser": {
    "main.d.ts": false,
    "main/fn/array.d.ts": "browser/fn/array.d.ts"
  }
}