Downloads the SQuAD training dataset

options

Options.

• :base_url - Dataset base URL.

  Defaults to "https://rajpurkar.github.io/SQuAD-explorer/dataset/"

• :train_dataset_file - Training set filename.

  Defaults to "train-v1.1.json"

• :cache_dir - Cache directory.

  Defaults to System.tmp_dir!()

examples

Examples

iex> Scidata.Squad.download()
[
  %{
    "paragraphs" => [
      %{
        "context" => "Architecturally, the school has a...",
        "qas" => [
          %{
            "answers" => [%{"answer_start" => 515, "text" => "Saint Bernadette Soubirous"}],
            "id" => "5733be284776f41900661182",
            "question" => "To whom did the..."
          }, ...
        ]
      }
    ],
    "title" => "University_of_Notre_Dame"
  }, ...
]

Downloads the SQuAD test dataset

options

Options.

• :base_url - Dataset base URL.

  Defaults to "https://rajpurkar.github.io/SQuAD-explorer/dataset/"

• :test_dataset_file - Test set filename.

  Defaults to "dev-v1.1.json"

• :cache_dir - Cache directory.

  Defaults to System.tmp_dir!()

examples

Examples

iex> Scidata.Squad.download_test()
[
  %{
    "paragraphs" => [
      %{
        "context" => "Super Bowl 50 was an American football game t...",
        "qas" => [
          %{
            "answers" => [
              %{"answer_start" => 177, "text" => "Denver Broncos"},...
            ],
            "id" => "56be4db0acb8001400a502ec",
            "question" => "Which NFL team represented the AFC at Super Bowl 50?"
          },
        ]
      }
    ],
    "title" => "Super_Bowl_50"
  }, ...
]

Convert result of download/0 or download_test/0 to map for use with Explorer.DataFrame.

examples

Examples

iex> columns_for_df = Scidata.Squad.download() |> Scidata.Squad.to_columns()
%{
  "answer_start" => [515, ...],
  "context" => ["Architecturally, the...", ...],
  "id" => ["5733be284776f41900661182", ...],
  "question" => ["To whom did the Vir...", ...],
  "answer_text" => ["Saint Bernadette Soubirous", ...],
  "title" => ["University_of_Notre_Dame", ...]
}
iex> Explorer.DataFrame.from_map(columns_for_df)
#Explorer.DataFrame<
[rows: 87599, columns: 6]
...
>