Skip to content

//> using file directives should allow URLs and git repository paths #1328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
benwbooth opened this issue Sep 7, 2022 · 2 comments · May be fixed by #3681
Open

//> using file directives should allow URLs and git repository paths #1328

benwbooth opened this issue Sep 7, 2022 · 2 comments · May be fixed by #3681
Labels
enhancement New feature or request spree using directives Issues tied with using directives.

Comments

@benwbooth
Copy link

It would be useful if the //> using file directives could accept URL and git repository paths in addition to local files. This would allow users to refer to their scala-cli module scripts from their executable scripts, no matter what paths the executable scripts were stored in:

//> using file "git+ssh://[email protected]/benwbooth/scala-scripts.git?dir=lib/scriptdir/my-script.sc"

or

//> using file "https://raw.githubusercontent.com/benwbooth/scala-scripts/lib/scriptdir/my-script.sc"

Deno has a similar feature for pulling in dependencies from URLs.
@Gedochao Gedochao added the enhancement New feature or request label Sep 8, 2022
@Gedochao Gedochao added the using directives Issues tied with using directives. label May 15, 2025
@Gedochao
Copy link
Contributor

Gedochao commented May 15, 2025 •
edited
Loading

Handling for remote sources is already in place for the command line, so this is a matter of porting this feature to the //> using file directive. Let me list some key points in the code for tackling this.

The standard entry point for inputs resolution is here:
https://github.com/VirtusLab/scala-cli/blob/main/modules/cli/src/main/scala/scala/cli/commands/shared/SharedOptions.scala#L688-L704

Where the interesting stuff starts here:

scala-cli/modules/build/src/main/scala/scala/build/input/Inputs.scala

Lines 319 to 333 in 4fb5c5b

private def forNonEmptyArgs(
args: Seq[String],
cwd: os.Path,
download: String => Either[String, Array[Byte]],
stdinOpt: => Option[Array[Byte]],
scriptSnippetList: List[String],
scalaSnippetList: List[String],
javaSnippetList: List[String],
markdownSnippetList: List[String],
acceptFds: Boolean,
forcedWorkspace: Option[os.Path],
enableMarkdown: Boolean,
allowRestrictedFeatures: Boolean,
extraClasspathWasPassed: Boolean
)(using invokeData: ScalaCliInvokeData): Either[BuildException, Inputs] = {

Inputs (including remote ones) are validated here:

scala-cli/modules/build/src/main/scala/scala/build/input/Inputs.scala

Lines 225 to 232 in 4fb5c5b

def validateArgs(
args: Seq[String],
cwd: os.Path,
download: String => Either[String, Array[Byte]],
stdinOpt: => Option[Array[Byte]],
acceptFds: Boolean,
enableMarkdown: Boolean
)(using programInvokeData: ScalaCliInvokeData): Seq[Either[String, Seq[Element]]] =

All remote (virtual) inputs are then represented as instances of scala.build.input.Virtual:

scala-cli/modules/build/src/main/scala/scala/build/input/Element.scala

Lines 16 to 57 in 4fb5c5b

sealed abstract class Virtual extends SingleElement {
def content: Array[Byte]
def source: String
def subPath: os.SubPath = {
val idx = source.lastIndexOf('/')
os.sub / source.drop(idx + 1)
}
def scopePath: ScopePath =
ScopePath(Left(source), subPath)
}
object Virtual {
val urlPathWithQueryParamsRegex = "https?://.*/([^/^?]+)(/?.*)?$".r
def apply(path: String, content: Array[Byte]): Virtual = {
val filename = path match {
case urlPathWithQueryParamsRegex(name, _) => name
case _ => path.split("/").last
}
val wrapperPath = os.sub / filename
if filename.endsWith(".scala") then VirtualScalaFile(content, path)
else if filename.endsWith(".java") then VirtualJavaFile(content, path)
else if filename.endsWith(".sc") then VirtualScript(content, path, wrapperPath)
else if filename.endsWith(".md") then VirtualMarkdownFile(content, path, wrapperPath)
else VirtualData(content, path)
}
}
sealed abstract class VirtualSourceFile extends Virtual {
def isStdin: Boolean = source.startsWith("<stdin>")
def isSnippet: Boolean = source.startsWith("<snippet>")
protected def generatedSourceFileName(fileSuffix: String): String =
if (isStdin) s"stdin$fileSuffix"
else if (isSnippet) s"${source.stripPrefix("<snippet>-")}$fileSuffix"
else s"virtual$fileSuffix"
}

The current implementation of the //> using file directive is defined here:

scala-cli/modules/directives/src/main/scala/scala/build/preprocessing/directives/Sources.scala

Lines 12 to 29 in 4fb5c5b

@DirectiveGroupName("Custom sources")
@DirectiveExamples("//> using file utils.scala")
@DirectiveUsage(
"`//> using file `_path_ | `//> using files `_path1_ _path2_ …",
"""`//> using file` _path_
|
|`//> using files` _path1_ _path2_ …
|""".stripMargin
)
@DirectiveDescription(
"Manually add sources to the project. Does not support chaining, sources are added only once, not recursively."
)
@DirectiveLevel(SpecificationLevel.SHOULD)
final case class Sources(
@DirectiveName("file")
files: DirectiveValueParser.WithScopePath[List[Positioned[String]]] =
DirectiveValueParser.WithScopePath.empty(Nil)
) extends HasBuildOptions {

Inputs from directives are then extracted here:

scala-cli/modules/build/src/main/scala/scala/build/CrossSources.scala

Lines 207 to 211 in 4fb5c5b

val sourcesFromDirectives =
preprocessedInputFromArgs
.flatMap(_.options)
.flatMap(_.internal.extraSourceFiles)
.distinct

cc @tgodzik

@ivan-klass
Copy link

Oh, I didn't know "using file" is non-recursive. Then it seems much simpler

@ivan-klass ivan-klass linked a pull request May 15, 2025 that will close this issue
Open
@Gedochao Gedochao linked a pull request May 16, 2025 that will close this issue
Support links in using file directive (implements #1328) #3681
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request spree using directives Issues tied with using directives.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support links in using file directive (implements #1328) ivan-klass/scala-cli
3 participants
@benwbooth @ivan-klass @Gedochao