Skip to content

Support links in using file directive (implements #1328) #3681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Support links in using file directive (implements #1328) #3681

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a86f8ca
Support links in using file directive
ivan-klass May 15, 2025
6a90821
remove git+ssh, add remote example
ivan-klass May 16, 2025
84c042a
Merge remote-tracking branch 'origin/main' into resolve-links-1328
ivan-klass May 16, 2025
d29327e
scalafix & updated docs
ivan-klass May 16, 2025
0691f65
New error class for Uri file errors
ivan-klass May 21, 2025
1964539
Merge remote-tracking branch 'origin/main' into resolve-links-1328
ivan-klass May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 38 additions & 7 deletions modules/build/src/main/scala/scala/build/CrossSources.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package scala.build

import java.io.File
import coursier.cache.FileCache
import coursier.util.{Artifact, Task}

import scala.build.CollectionOps.*
import scala.build.EitherCps.{either, value}
Expand All @@ -11,7 +12,8 @@ import scala.build.errors.{
CompositeBuildException,
ExcludeDefinitionError,
MalformedDirectiveError,
Severity
Severity,
UsingFileFromUriError
}
import scala.build.input.ElementsUtils.*
import scala.build.input.*
Expand Down Expand Up @@ -209,7 +211,8 @@ object CrossSources {
.flatMap(_.options)
.flatMap(_.internal.extraSourceFiles)
.distinct
val inputsElemFromDirectives: Seq[SingleFile] =

val inputsElemFromDirectives: Seq[SingleElement] =
value(resolveInputsFromSources(sourcesFromDirectives, inputs.enableMarkdown))
val preprocessedSourcesFromDirectives: Seq[PreprocessedSource] =
value(preprocessSources(inputsElemFromDirectives.pipe(elements =>
Expand Down Expand Up @@ -403,7 +406,37 @@ object CrossSources {
fromInputs ++ fromSources ++ fromSourcesWithRequirements
}

private def resolveInputsFromSources(sources: Seq[Positioned[os.Path]], enableMarkdown: Boolean) =
// TODO: reuse existing one? e.g. scala.cli.commands.shared.SharedOptions.coursierCache
lazy val fileCache: FileCache[coursier.util.Task] = FileCache()
Comment on lines +409 to +410
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, I think it's accurate that SharedOptions.coursierCache should be used.

Copy link
Author

@ivan-klass ivan-klass May 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Gedochao but scala.cli.commands.shared.SharedOptions.coursierCache is in cli module, while we're in build here - so that definition is inaccessible (build doesn't depend on cli)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's why we haven't picked it up and left a todo

Copy link
Contributor

@Gedochao Gedochao May 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...well, yes, but cli depends on build.
so what I mean here is that the way to go would be to pass it all the way from cli to CrossSources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you just leave a TODO, it's very likely to get forgotten.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you don't have time to fix it, please at least create a ticket for the follow-up and link it back in the code.


private def downloadFile(pUri: Positioned[java.net.URI]) =
import scala.build.options.ScalaVersionUtil.fileWithTtl0
val artifact = Artifact(pUri.value.toString).withChanging(true)
fileCache.fileWithTtl0(artifact)
.left
.map(cause => new UsingFileFromUriError(pUri.value, pUri.positions, cause))
.map(f => os.read.bytes(os.Path(f, Os.pwd))).map(content =>
Seq(Virtual(pUri.value.toString, content))
)

type CodeFile = os.Path | java.net.URI

private def resolveInputsFromSources(
sources: Seq[Positioned[CodeFile]],
enableMarkdown: Boolean
) =
val links = sources.collect {
case Positioned(pos, value: java.net.URI) => Positioned(pos, value)
}
val paths = sources.collect {
case Positioned(pos, value: os.Path) => Positioned(pos, value)
}

(resolveInputsFromPath(paths, enableMarkdown) ++ links.map(downloadFile)).sequence
.left.map(CompositeBuildException(_))
.map(_.flatten)

private def resolveInputsFromPath(sources: Seq[Positioned[os.Path]], enableMarkdown: Boolean) =
sources.map { source =>
val sourcePath = source.value
lazy val dir = sourcePath / os.up
Expand All @@ -424,9 +457,7 @@ object CrossSources {
else s"$sourcePath: not found path defined in using directive."
Left(new MalformedDirectiveError(msg, source.positions))
}
}.sequence
.left.map(CompositeBuildException(_))
.map(_.flatten)
}

/** Filters out the sources from the input sequence based on the provided 'exclude' patterns. The
* exclude patterns can be absolute paths, relative paths, or glob patterns.
Expand Down
12 changes: 12 additions & 0 deletions modules/directives/src/main/scala/scala/build/errors/UsingFileFromUriError.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package scala.build.errors

import java.net.URI

import scala.build.Position

final class UsingFileFromUriError(uri: URI, positions: Seq[Position], cause: Throwable)
extends BuildException(
message = s"Error using file from $uri - ${cause.getLocalizedMessage}",
positions = positions,
cause = cause
)
19 changes: 18 additions & 1 deletion modules/directives/src/main/scala/scala/build/preprocessing/directives/Sources.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ import scala.util.Try

@DirectiveGroupName("Custom sources")
@DirectiveExamples("//> using file utils.scala")
@DirectiveExamples(
"//> using file https://raw.githubusercontent.com/softwaremill/sttp/refs/heads/master/examples/src/main/scala/sttp/client4/examples/json/GetAndParseJsonCatsEffectCirce.scala"
)
@DirectiveUsage(
"`//> using file `_path_ | `//> using files `_path1_ _path2_ …",
"""`//> using file` _path_
Expand All @@ -27,6 +30,17 @@ final case class Sources(
files: DirectiveValueParser.WithScopePath[List[Positioned[String]]] =
DirectiveValueParser.WithScopePath.empty(Nil)
) extends HasBuildOptions {

private def codeFile(codeFile: String, root: os.Path): Sources.CodeFile =
scala.util.Try {
val uri = java.net.URI.create(codeFile)
uri.getScheme match {
case "file" | "http" | "https" => uri
}
}.getOrElse {
os.Path(codeFile, root)
}

def buildOptions: Either[BuildException, BuildOptions] = either {

val paths = files
Expand All @@ -35,7 +49,7 @@ final case class Sources(
for {
root <- Directive.osRoot(files.scopePath, positioned.positions.headOption)
path <- {
try Right(positioned.map(os.Path(_, root)))
try Right(positioned.map(codeFile(_, root)))
catch {
case e: IllegalArgumentException =>
Left(new WrongSourcePathError(positioned.value, e, positioned.positions))
Expand All @@ -55,5 +69,8 @@ final case class Sources(
}

object Sources {

type CodeFile = os.Path | java.net.URI

val handler: DirectiveHandler[Sources] = DirectiveHandler.derive
}
4 changes: 3 additions & 1 deletion modules/options/src/main/scala/scala/build/options/InternalOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ import scala.build.errors.BuildException
import scala.build.interactive.Interactive
import scala.build.interactive.Interactive.InteractiveNop

type CodeFile = os.Path | java.net.URI

final case class InternalOptions(
keepDiagnostics: Boolean = false,
cache: Option[FileCache[Task]] = None,
Expand All @@ -24,7 +26,7 @@ final case class InternalOptions(
* really needed.
*/
keepResolution: Boolean = false,
extraSourceFiles: Seq[Positioned[os.Path]] = Nil,
extraSourceFiles: Seq[Positioned[CodeFile]] = Nil,
exclude: Seq[Positioned[String]] = Nil,
offline: Option[Boolean] = None
) {
Expand Down
2 changes: 2 additions & 0 deletions website/docs/reference/directives.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,8 @@ Manually add sources to the project. Does not support chaining, sources are adde
#### Examples
`//> using file utils.scala`

`//> using file https://raw.githubusercontent.com/softwaremill/sttp/refs/heads/master/examples/src/main/scala/sttp/client4/examples/json/GetAndParseJsonCatsEffectCirce.scala`
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Gedochao I think it would be nice to reference a file within scala-cli repo itself

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean... dunno if it matters, as long as the example is useful, to be honest. 🤷


### Dependency

Add dependencies
Expand Down
2 changes: 2 additions & 0 deletions website/docs/reference/scala-command/directives.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,8 @@ Manually add sources to the project. Does not support chaining, sources are adde
#### Examples
`//> using file utils.scala`

`//> using file https://raw.githubusercontent.com/softwaremill/sttp/refs/heads/master/examples/src/main/scala/sttp/client4/examples/json/GetAndParseJsonCatsEffectCirce.scala`

### Exclude sources

Exclude sources from the project
Expand Down
Loading