Creating a complex web-service to share files within my household.
While discussions around programming and software development are often relegated to syntax and programming itself, the requirements of software roles often far exceed the bounds of simply programming and understanding code. For example, for any software that is written there is documentation. All of this information is stored in files, and being able to organize and manipulate those files are also crucial to getting anything published as a developer. We could have the best project in the world but a simple file mistake could mess it up. Given how sensitive and important files in a programming project typically are, it is important to keep these files organized. As developers, we all eventually develop our own workflows and techniques to work with this aspect of our projects.
One thing that has revolutionized my workflow as a developer is centralizing everything. From any computer I can access, I am able to call home and access my dev folder using ssh . When I am at home, for example, I may be developing on the couch, using the computer on the desk remotely. There are huge advantages to doing things remotely. No matter what hardware I am using, I am able to access my files — if my back hurts, I don’t have to stop working I can just use a laptop. While using SSH and SSHFS works for me, there is a major short-coming to my workflow.. It works exclusively for me.
I am not the only computer nerd in my household, and when that is the case there are of course times where we might want to transfer a file between us. This is only exemplified if you are working on something together, or want to share a library of media with each-other — while I can SSH into my own system just fine, sometimes I want people to be able to access files locally without my intervention or having to SSH as me. Rather than a centralized development computer, I want centralized Network- Attached-Storage.
While there are plenty of network-attached-storage solutions on the software side, ranging from simple command-line tools to entire GNU+Linux operating-systems, as a developer I am constantly seeking to try new things and I always thought this would be an interesting project. Projects that I make for myself like this present me with a great opportunity to scope something out on my own. This experience, in turn benefits me in future endeavors whether alone or working as a team.
source
Building A NAS Server
With this Network-Attached-Storage, I wanted to do something unique. My application consists of two different components working together. Firstly, there is an API — an HTTP server that I will have running on a physical server eventually, holding a plethora of harddrives. To communicate with this API, I plan to use a GUI application on client machines. This client will be used to connect to the server. As long-term readers will likely expect, I am going to be using the Toolips web-development framework to handle the web-based API. While it of course crossed my mind that we could use our own packet structure and the UDP protocol to speed things up, my plan for creating the application later is going to fare a lot better if we use HTTP and TCP for a connection-driven protocol right now. Ultimately, while we might need to translate a lot of data through this protocol — which will travel at web-download speed, we are not sending a lot of requests consistently and the number of clients is pretty low. Considering this, I think it makes a lot more sense to make a conventional Toolips server, rather than a ToolipsUDP server, as the connection protocol will benefit a lot in terms of sending messages back and forth. The great thing about using a protocol with a connection in this case is that we don’t need to communicate when information is sent or received and we can recieve a response without requiring an additional server for the client.
using Toolips
Toolips.new_app("ChiNAS")
type system
My favorite thing about building projects with Toolips is that the front-end and the back-end are combined into the same application, and everything is situated in Julia — so we are able to use Julia’s paradigm and programming techniques to our advantage. As a result, for this type of back-end heavy project I typically start with normal Julia constructors. Holding data inside of a type makes it far more organized and easier to work with, and may also give us some indication as to the direction or scope of our project.
module ChiNAS
using Toolips
using TOML
using REPLMaker
using Toolips.Components
CONNECTED = "":0
# types
abstract type AbstractRepository end
mutable struct Repository <: AbstractRepository
uri::String
file_count::Int64
end
mutable struct NASUser
ip::String
name::String
wd::String
end
mutable struct NASManager <: Toolips.AbstractExtension
hostname::String
home_dir::String
repos::Vector{AbstractRepository}
users::Vector{NASUser}
secret::String
end
Here I have a basic version of three types. First, we have the abstract-type AbstractRepository and its concrete counter-part the Repository , a sub-type of AbstractRepository . The goal of this new type is to provide cloning and updating capabilities to specific directories of files — helping to give an extra element of organization to the final result, while also making changes to those files easier to manage. In short, this aspect of the project will allow me to let other people on my network view and edit my files on a project-to-project basis. We might not get to fully implementing this feature today, but this is planned to be one of the many features that make this project unique.
The next type is the NASUser . This saves the user’s IPv4 address, the name the corresponding client is known as by the server, and finally their current working directory in the server’s file-system — wd . This type will allow us to easily track and store active users.
Finally, there is the server extension NASManager . This server extension will be loaded into our Connection , allowing us to access all of this data as necessary. In Toolips 0.3 extensions had changed somewhat dramatically, in this form of Toolips we need to export our extensions, so they are always defined as a field of our server’s Module :
MANAGER = NASManager("", "", Vector{AbstractRepository}(), Vector{NASUser}(), "testpass")
We may come back to some of these types and add a few new fields to them — we will likely need to create new fields to manage more information about repositories. For now, we will move onto utilizing these types in a response and getting a basic version of our server running.
building the API
At its core, our project revolves around an API that serves our clients information about our server’s file-system. To build this API, I will start by creating a simple function to start the server, host .
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
end
We are also going to need a route:
main = route("/") do c::Toolips.AbstractConnection
end
Our request system will start with a step-by-step handshake:
- â„–1: On initial request, the client will send us a
secretas a GET request argument. We will verify thissecretbefore granting access to the server. - â„–2: Upon verification of the
secret, the client will send the user’sname, which the user is prompted to enter. The server simply stores this information and responds that it was successful in doing so. - №3: After all of this has been performed, a blank request can be made to the server — which will act as a “ list directory.” With this, we will add more commands, such as the ability to change the working directory or download files.
Beginning in our main route that was just created, here is my first conditional — attempting to determine whether or not the :secret is provided:
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(get_ip(c), "", "~/")
push!(MANAGER.users, new_user)
write!(c, "confirmed")
return
end
write!(c, "denied")
end
After checking the arguments for the secret, we get the secret and verify it is the secret. This completes the minimum viable step â„–1, step number two is also simple, but will require us to expand on the NASUser to more easily access this data:
import Base: in, getindex
function in(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
~(isnothing(f))::Bool
end
function getindex(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
vec[f]::NASUser
end
And now we add this to our route:
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
client_ip::String = get_ip(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(client_ip, "", "~/")
push!(MANAGER.users, new_user)
write!(c, "confirmed")
return
end
write!(c, "denied")
elseif :name in args and client_ip in MANAGER.users
MANAGER.users[client_ip].name = args[name]
write!(c, "success")
end
elseif :name in args and client_ip in MANAGER.users
I have been writing too much Python.
elseif :name in args && client_ip in MANAGER.users
After all of this, we will verify a user has been authenticated before finally serving them with the actual API:
ips = [user.ip for user in MANAGER.users]
if ~(client_ip in ips)
write!(c, "secret?")
return
end
We finish by getting the user, for now:
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
client_ip::String = get_ip(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(client_ip, "", "~/")
push!(MANAGER.users, new_user)
write!(c, "confirmed")
return
end
write!(c, "denied")
elseif :name in args && client_ip in MANAGER.users
MANAGER.users[client_ip].name = args[name]
write!(c, "success")
end
ips = [user.ip for user in MANAGER.users]
if ~(client_ip in ips)
write!(c, "secret?")
return
end
user = MANAGER.users[client_ip]
end
Back in our host function, we are going to be reading a configuration file to get the information on the current server we are trying to work with. This configuration information will then of course be loaded into our NASManager type. I will start our host function by attempting to find this config.toml file.
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
# read the path, make config, build the routes
path = replace(path, "\\" => "/")
dir_read::Vector{String} = readdir(path)
if ~("config.toml" in dir_read)
config_path::String = path * "/config.toml"
touch(config_path)
open(config_path, "w") do o::IO
end
end
First, I make the provided path — defaulted as the working directory —and replace any backslashes with "/" by using replace. This might seem insignificant, but actually ensures that this Julia software is platform agnostic. More specifically, pwd in Julia on a Windows system will return a path like C:\\Users/emmac . That path is valid on Windows, not only that but \ and / on Windows, at least in the context of this main drive (I don’t use Windows) are interchangeable — so C:\\ is also C:// — but the file-system is going to call it C:\\ , so to make things easier for us as developers we can convert it to C:// to make it consistent with every other file-system we are developing software for. Of course, we split and join by / , so we want this to be consistent!
Next, we check if config.toml is in the path provided. If not, we go ahead and touch it before opening to write it. Now we can go ahead and use TOML to parse that configuration file.
config = TOML.parse(read(path * "/config.toml", String))
We won’t create the code for this yet, but here we will also loop through the users, assigning the return to the MANAGER.users field.
MANAGER.users = [begin
end for user in config["users"]]
We are going to finish this function off, for now, by adding the secret . I am going to be managing this secret primarily by entering it into the file. In other words, the secret can be customized; the plan is to only serve files through a completely customized client via GET requests, so we will not need to worry about any clients in this NAS getting access to it. Furthermore, a Connection without the secret or verification is refused in the first place — and my intention is not to deploy ChiNAS publically in any capacity, but to use it locally for my household.
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
# read the path, make config, build the routes
path = replace(path, "\\" => "/")
dir_read::Vector{String} = readdir(path)
if ~("config.toml" in dir_read)
config_path::String = path * "/config.toml"
touch(config_path)
open(config_path, "w") do o::IO
end
end
config = TOML.parse(read(path * "/config.toml", String))
MANAGER.users = [begin
end for user in config["users"]]
# start the server, add the routes
start!(ChiNAS, ip:port)
println("ChiNAS is now active!")
# key
secret::String = gen_ref(5)
secret_path::String = path * "/secret.txt"
println("secret key: ", secret)
if ~("secret.txt" in dir_read)
touch(secret_path)
else
secret = read(secret_path, String)
end
open(path * "/secret.txt", "w") do o::IOStream
write(o, secret)
end
end
We will be returning to this function soon.
serving files
For serving files, I wanted to do things a bit differently than one might expect. Through our code, we have already defined somewhat of a premise for a NAS directory. So far, this directory will contain config.toml , as well as secret.txt . The file-system for the storage will actually sit below this in a new folder called home . This folder will contain all of the regular files on the NAS. I will also make a new folder called repositories , which will specifically hold files in repositories.
While the eventual plan is to create a more complex GUI, for now I am going to make a simple CLI to come with the package. Some might have noticed that I used REPLMaker at the top of this project — we are going to be using REPLMaker to create a quick command-line interface to test our handshake system. This will also give us code to replicate later on when we make a connecting application. We will start by requesting the server to ensure it is up. We should get secret? in response if we do not provide a secret.
function connect(ip::String, port::Int64; path::String = pwd())
init_response::String = Toolips.get(ip:port)
if init_response == "secret?"
println("This storage requires a password to access.")
println("Please enter the access password:")
pwd = readline()
second_response = Toolips.get("http://$ip:$port/?secret=$pwd")
if second_response == "denied"
println("Access denied, it seems the password has been entered wrong.")
return(connect(ip, port, path = path))
end
println("Select a user-name from which to access this remote file-system:")
name = readline()
third_response = Toolips.get("http://$ip:$port/?name=$name")
if ~(third_response == "success")
print("failure. name taken? for now we give up here.")
return
end
return(connect(ip, port, path = path))
end
With this, we will prompt the user for the secret. After this, the server wants us to set a user-name so we will do so. Finally, the function will actually recall itself in the return — now that the secret handshake has been performed. When this call back to the function happens, init_response == "secret" will be false , leading to the code below it being executed. Here we will set up the REPL with initrepl .
server_name = Toolips.get("http://$ip:$port/hostname")
CONNECTED = ip:port
interpret_file_response(init_response)
initrepl(send_to_connected,
prompt_text="$server_name >",
prompt_color=:lightblue,
start_key="-",
mode_name="Remote Filesystem")
end
This CONNECTED is a new global variable which tells us if the application is currently connected to a server.
The send_to_connected is a new function we now need to create as a handler for the REPL.
function send_to_connected(line::String)
split_cmd::Vector{SubString} = split(line, " ")
if split_cmd[1] == "download"
download_url = Toolips.post(CONNECTED, "DOWNLOAD;$(split_cmd[2])")
return
end
response = Toolips.post(CONNECTED, replace(line, " " => ";"))
end
Here we Toolips.post directly to CONNECTED . Notice how I made an exception for if the first command is downloaded, as this is going to require a different protocol. Here I post for a download link, we will eventually write a generated (one-time) download link for these requests on the server-side. This all happens back in our main route:
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
client_ip::String = get_ip(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(client_ip, "", "~/")
push!(MANAGER.users, new_user)
write!(c, "confirmed")
return
end
write!(c, "denied")
elseif :name in args && client_ip in MANAGER.users
MANAGER.users[client_ip].name = args[name]
write!(c, "success")
end
ips = [user.ip for user in MANAGER.users]
if ~(client_ip in ips)
write!(c, "secret?")
return
end
user = MANAGER.users[client_ip]
end
Now that we have the user , we are going to grab the post and determine the command. We will start by grabbing the post:
Referring back to our original function, our first split will be the command and subsequent splits will be arguments. If there is no command or the command is ls , we will list directories.
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
client_ip::String = get_ip(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(client_ip, "", "~/")
push!(MANAGER.users, new_user)
write!(c, "confirmed")
return
end
write!(c, "denied")
elseif :name in args && client_ip in MANAGER.users
MANAGER.users[client_ip].name = args[name]
write!(c, "success")
end
ips = [user.ip for user in MANAGER.users]
if ~(client_ip in ips)
write!(c, "secret?")
return
end
user = MANAGER.users[client_ip]
request = get_post(c)
command_split = split(request, ";")
end
From here, there are a lot of different approaches we could use to determine which command is being used. The most obvious is a conditional, where we ask if the command is X or Y and do the appropriate action. A neat trick in Julia, however, will allow us to make a superior design using the type system. I am going to start by making a simple type that only holds a parameter.
mutable struct NASCommand{T <: Any} end
user = MANAGER.users[client_ip]
request = get_post(c)
command_split = split(request, ";")
f = findfirst(";", request)
if length(command_split) == 1 || isnothing(f) || command_split[1] == "ls"
end
do_command(user, NASCommand{Symbol(command_split[1])}(), command_split[2:length(command_split)] ...)
end
We will want to write the return of this do_command to the Connection :
write!(c,
do_command(user, NASCommand{Symbol(command_split[1])}(),
command_split[2:length(command_split)] ...))
end
Now we need to make a do_command function.
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
end
Let’s go ahead and add the cd and ls commands. For cd , we will want to first read check if the cd is to .. .
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
user.wd = join(wdsplit[1:length(wdsplit) - 1], "/")
end
end
end
If the working directory is not the top of the home, we will change directory up. If .. is not the request directory, we will want to first check if the directory exists in our currently selected directory. If the directory exists, we will CD to it:
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
user.wd = join(wdsplit[1:length(wdsplit) - 1], "/")
end
end
current_dir_files = readdir(replace(user.wd, "~/" => NASManager.home_dir * "/"))
if ~(selected in current_dir_files)
return("ERROR: Directory does not exist to change into.")
end
user.wd = user.wd * selected
end
Finally, we will add the listing of directories to our main route:
command_split = split(request, ";")
f = findfirst(";", request)
if length(command_split) == 1 || isnothing(f) || command_split[1] == "ls"
current_dir_files = readdir(replace(user.wd, "~/" => NASManager.home_dir * "/"))
write!(c, join([filename for filename in current_dir_files], ";"))
return
end
write!(c,
do_command(user, NASCommand{Symbol(command_split[1])}(),
command_split[2:length(command_split)] ...))
end
The response I write sends back the file-names separated by ; . I left this a bit open-ended, as it is likely I am going to want to add more data here in the future — such as whether or not each name is a file or a directory. For now, we will go back to the client and split these incoming file names before printing them. For now, we will simply use replace to make them \ns :
function send_to_connected(line::String)
split_cmd::Vector{SubString} = split(line, " ")
if split_cmd[1] == "download"
download_url = Toolips.post(CONNECTED, "DOWNLOAD:$(split_cmd[2])")
return
end
response = Toolips.post(CONNECTED, replace(line, " " => ";"))
println(replace(response, ";" => "\n"))
end
The last thing we need to do to get this up and running is actually load some users from our configuration file. We will also fill in the default for if one is not created.
if ~("config.toml" in dir_read)
config_path::String = path * "/config.toml"
touch(config_path)
basic_dct = Dict("users" => Dict("admin" => Dict("ip" => "", "wd" => "~/")))
open(config_path, "w") do o::IO
TOML.print(o, basic_dct)
end
end
config = TOML.parse(read(path * "/config.toml", String))
# get user and repo data
MANAGER.users = [begin
info = config["users"][user]
NASUser(info["ip"], user, info["wd"])
end for user in config["users"]]
This should work to some degree now, though the code is untested. A few tweaks might need to be made, but here is the current state of the server module:
module ChiNAS
using Toolips
using TOML
using REPLMaker
using Toolips.Components
import Base: in, getindex
CONNECTED = "":0
abstract type AbstractRepository end
mutable struct Repository <: AbstractRepository
uri::String
file_count::Int64
end
mutable struct NASUser
ip::String
name::String
wd::String
end
mutable struct NASManager <: Toolips.AbstractExtension
hostname::String
home_dir::String
repos::Vector{AbstractRepository}
users::Vector{NASUser}
secret::String
end
mutable struct NASCommand{T <: Any} end
function in(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
~(isnothing(f))::Bool
end
function getindex(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
vec[f]::NASUser
end
logger = Toolips.Logger()
MANAGER = NASManager("", "", Vector{AbstractRepository}(), Vector{NASUser}(), "testpass")
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
path = replace(path, "\\" => "/")
dir_read::Vector{String} = readdir(path)
if ~("config.toml" in dir_read)
config_path::String = path * "/config.toml"
touch(config_path)
open(config_path, "w") do o::IO
end
end
config = TOML.parse(read(path * "/config.toml", String))
MANAGER.users = [begin
end for user in config["users"]]
start!(ChiNAS, ip:port)
println("ChiNAS is now active!")
secret::String = gen_ref(5)
secret_path::String = path * "/secret.txt"
println("secret key: ", secret)
if ~("secret.txt" in dir_read)
touch(secret_path)
else
secret = read(secret_path, String)
end
open(path * "/secret.txt", "w") do o::IOStream
write(o, secret)
end
end
function connect(ip::String, port::Int64; path::String = pwd())
init_response::String = Toolips.get(ip:port)
if init_response == "secret?"
println("This storage requires a password to access.")
println("Please enter the access password:")
pwd = readline()
second_response = Toolips.get("http://$ip:$port/?secret=$pwd")
if second_response == "denied"
println("Access denied, it seems the password has been entered wrong.")
return(connect(ip, port, path = path))
end
println("Select a user-name from which to access this remote file-system:")
name = readline()
third_response = Toolips.get("http://$ip:$port/?name=$name")
if ~(third_response == "success")
print("failure. name taken? for now we give up here.")
return
end
return(connect(ip, port, path = path))
end
server_name = Toolips.get("http://$ip:$port/hostname")
CONNECTED = ip:port
interpret_file_response(init_response)
initrepl(send_to_connected,
prompt_text="$server_name >",
prompt_color=:lightblue,
start_key="-",
mode_name="Remote Filesystem")
end
function interpret_file_response()
end
function send_to_connected(line::String)
split_cmd::Vector{SubString} = split(line, " ")
if split_cmd[1] == "download"
download_url = Toolips.post(CONNECTED, "DOWNLOAD:$(split_cmd[2])")
return
end
response = Toolips.post(CONNECTED, replace(line, " " => ";"))
println(replace(response, ";" => "\n"))
end
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
client_ip::String = get_ip(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(client_ip, "", "~/")
push!(MANAGER.users, new_user)
write!(c, "confirmed")
return
end
write!(c, "denied")
elseif :name in args && client_ip in MANAGER.users
MANAGER.users[client_ip].name = args[name]
write!(c, "success")
end
ips = [user.ip for user in MANAGER.users]
if ~(client_ip in ips)
write!(c, "secret?")
return
end
user = MANAGER.users[client_ip]
request = get_post(c)
command_split = split(request, ";")
f = findfirst(";", request)
if length(command_split) == 1 || isnothing(f) || command_split[1] == "ls"
current_dir_files = readdir(replace(user.wd, "~/" => NASManager.home_dir * "/"))
write!(c, join([filename for filename in current_dir_files], ";"))
return
end
write!(c,
do_command(user, NASCommand{Symbol(command_split[1])}(),
command_split[2:length(command_split)] ...))
end
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
user.wd = join(wdsplit[1:length(wdsplit) - 1], "/")
end
end
current_dir_files = readdir(replace(user.wd, "~/" => NASManager.home_dir * "/"))
if ~(selected in current_dir_files)
return("ERROR: Directory does not exist to change into.")
end
user.wd = user.wd * selected * "/"
end
export main, default_404, logger, MANAGER
end
emmac | clara 02:42:22 > cd ChiNAS
emmac | clara 02:42:24 > ls
dev.jl Manifest.toml Project.toml src
emmac | clara 02:42:25 > mkdir test
emmac | clara 02:42:34 > cd test
emmac | clara 02:42:40 > mkdir home
emmac | clara 02:43:00 > mkdir repositories
emmac | clara 02:43:04 > tree .
.
├── home
└── repositories
3 directories, 0 files
emmac | clara 02:43:06 > julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _\` | |
| | |_| | | | (_| | | Version 1.10.3 (2024-04-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia>
ChiNAS) pkg> activate ../.
Activating project at \`~/dev/packages/chifi/ChiNAS\`
julia> using ChiNAS
Precompiling ChiNAS
Info Given ChiNAS was explicitly requested, output will be shown live
ERROR: LoadError: ArgumentError: Package ChiNAS does not have TOML in its dependencies:
- You may have a partially installed environment. Try \`Pkg.instantiate()\`
to ensure all packages in the environment are installed.
- Or, if you have ChiNAS checked out for development and have
added TOML as a dependency but haven
environment
- Otherwise you may need to report an issue with ChiNAS
After actually adding our packages, I realized “ REPLMaker “ is “ ReplMaker .“
julia> using ChiNAS
Precompiling ChiNAS
1 dependency successfully precompiled in 1 seconds. 28 already precompiled.
With that quick fix, it precompiles. Now to determine if it actually works. We will begin by starting a server. I had to fix this first:
MANAGER.users = [begin
info = config["users"][user]
NASUser(info["ip"], user, info["wd"])
end for user in keys(config["users"])]
For whatever reason, I thought we were looping the keys — this was fixed by of course actually looping the keys. This time, the server started — but we did get an error while writing the secret key.
julia> ChiNAS.host("127.0.0.1", 8000)
🌷 toolips> loaded router type: Vector{Toolips.Route{Toolips.AbstractConnection}}
🌷 toolips> server listening at http:
Active manifest files: 15 found
Active artifact files: 4 found
Active scratchspaces: 2 found
Deleted no artifacts, repos, packages or scratchspaces
ChiNAS is now active![ Info: Listening on: 127.0.0.1:8000, thread id: 1
ERROR: UndefVarError: \`gen_ref\` not defined
Stacktrace:
[1] host(ip::String, port::Int64; path::String, hostname::String)
@ ChiNAS ~/dev/packages/chifi/ChiNAS/src/ChiNAS.jl:71
[2] host(ip::String, port::Int64)
@ ChiNAS ~/dev/packages/chifi/ChiNAS/src/ChiNAS.jl:49
[3] top-level scope
@ REPL[3]:1
gen_ref is not exported, so we will need to call it from Toolips or import it.
(@v1.10) pkg> activate ../.
Activating project at \`~/dev/packages/chifi/ChiNAS\`
julia> using ChiNAS
Precompiling ChiNAS
1 dependency successfully precompiled in 2 seconds. 28 already precompiled.
julia> ChiNAS.host("127.0.0.1", 8000)
🌷 toolips> loaded router type: Vector{Toolips.Route{Toolips.AbstractConnection}}
🌷 toolips> server listening at http://127.0.0.1:8000
Active manifest files: 15 found
Active artifact files: 4 found
Active scratchspaces: 2 found
Deleted no artifacts, repos, packages or scratchspaces
ChiNAS is now active![ Info: Listening on: 127.0.0.1:8000, thread id: 1
secret key: dhjkt
5
Now let’s give the client-side a try using our connect function.
connect(ip::String, port::Int64; path::String = pwd())
julia> ChiNAS.connect("127.0.0.1", 8000)
┌ Error: handle_connection handler error.
│
│ ===========================
│ HTTP Error message:
│
│ ERROR: AbstractDict collections only contain Pairs;
│ Either look for e.g. A=>B instead, or use the \`keys\` or \`values\`
│ function if you are looking for a key or value respectively.
│ Stacktrace:
│ [1] error(s::String)
│ @ Base ./error.jl:35
│ [2] in(p::Symbol, a::Dict{Symbol, String})
│ @ Base ./abstractdict.jl:28
│ [3] (::ChiNAS.var"#13#15")(c::Toolips.Connection)
│ @ ChiNAS ~/dev/packages/chifi/ChiNAS/src/ChiNAS.jl:141
Appears I messed up somewhere.
write!(c, "denied")
elseif :name in args && client_ip in MANAGER.users
MANAGER.users[client_ip].name = args[name]
write!(c, "success")
end
If :name in keys(args) . Funnily enough, the conditional above it is done correctly.
julia> ChiNAS.connect("127.0.0.1", 8000)
This storage requires a password to access.
Please enter the access password:
whiht
Select a user-name from which to access this remote file-system:
admin
failure. name taken? for now we give up here.
Right, so I forgot — my original intention was for this to be the creation of new names. However, I no longer want this to be the case so we will simply update the IP of a user if they are already found.
elseif :name in keys(args) && client_ip in MANAGER.users
name::String = args[:name]
f = findfirst(user -> user.name == name, MANAGER.users)
if isnothing(f)
MANAGER.users[client_ip].name = name
else
MANAGER.users[f].ip = client_ip
end
write!(c, "success")
return
end
It turns out another thing we forgot to do was set the MANAGER.secret after the server is started.
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
# read the path, make config, build the routes
path = replace(path, "\\" => "/")
dir_read::Vector{String} = readdir(path)
if ~("config.toml" in dir_read)
config_path::String = path * "/config.toml"
touch(config_path)
basic_dct = Dict("users" => Dict("admin" => Dict("ip" => "", "wd" => "~/")))
open(config_path, "w") do o::IO
TOML.print(o, basic_dct)
end
end
config = TOML.parse(read(path * "/config.toml", String))
# get user and repo data
MANAGER.users = [begin
info = config["users"][user]
NASUser(info["ip"], user, info["wd"])
end for user in keys(config["users"])]
# start the server, add the routes
start!(ChiNAS, ip:port)
# key
secret::String = Toolips.gen_ref(5)
secret_path::String = path * "/secret.txt"
if ~("secret.txt" in dir_read)
touch(secret_path)
else
secret = read(secret_path, String)
end
open(path * "/secret.txt", "w") do o::IOStream
write(o, secret)
end
println("secret key: ", secret)
MANAGER.secret = secret
end
With this, we are officially able to login to the server.
julia> ChiNAS.connect("127.0.0.1", 8000)
[""]
This storage requires a password to access.
Please enter the access password:
dhjkt
Select a user-name from which to access this remote file-system:
admin
┌ Error: handle_connection handler error.
│
│ ===========================
│ HTTP Error message:
│
│ ERROR: type DataType has no field home_dir
│ Stacktrace:
│ [1] getproperty
│ @ ./Base.jl:32 [inlined]
│ [2] (::ChiNAS.var"#13#16")(c::Toolips.Connection)
│ @ ChiNAS ~/dev/packages/chifi/ChiNAS/src/ChiNAS.jl:167
│ [3] route!(c::Toolips.Connection, r::Toolips.Route{Toolips.AbstractConnection})
│ @ Toolips ~/.julia/packages/Toolips/ae5k0/src/core.jl:942
│ [4] route!(c::Toolips.Connection, tr::Vector{Toolips.Ro
However, we get an immediate error. It seems, just from reading the code not the error, we likely called the type NASManager somewhere instead of MANAGER. Whoops. We did this in our two recent command calls for listing the directories and changing them, the only place we really call for the field home_dir .
With that issue fixed, I ran into a few other minor issues, but eventually I was able to get the project working as intended:
...
Please enter the access password:
dhjkt
Select a user-name from which to access this remote file-system:
admin
REPL mode Remote Filesystem initialized. Press - to enter and backspace to exit.
"Prompt(\"127.0.0.1:8000 >\",...)"
127.0.0.1:8000 >ls
afs
bin
boot
dev
etc
home
lib
lib64
lost+found
media
mnt
opt
proc
root
run
sbin
snap
srv
sys
tmp
usr
var
Problematically, this is the / directory on my computer. Fortunately, I think this is because we never actually set the project up to actually load the home directory. We will of course be doing so in host.
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
# read the path, make config, build the routes
path = replace(path, "\\" => "/")
MANAGER.home_dir = path * "/"
dir_read::Vector{String} = readdir(path)
if ~("config.toml" in dir_read)
julia> ChiNAS.connect("127.0.0.1", 8000)
This storage requires a password to access.
Please enter the access password:
dhjkt
Select a user-name from which to access this remote file-system:
admin
REPL mode Remote Filesystem initialized. Press - to enter and backspace to exit.
"Prompt(\"127.0.0.1:8000 >\",...)"
127.0.0.1:8000 >ls
config.toml
home
repositories
secret.txt
127.0.0.1:8000 >
Right, it is supposed to be pwd * home .
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
# read the path, make config, build the routes
path = replace(path, "\\" => "/")
MANAGER.home_dir = path * "/" * "home/"
Since we did this wrong in this case, this will be a great opportunity to test if we can change directories.
127.0.0.1:8000 >ls
config.toml
home
repositories
secret.txt
127.0.0.1:8000 >cd home
~/home/
127.0.0.1:8000 >ls
┌ Error: handle_connection handler error.
│
│ ===========================
│ HTTP Error message:
│
│ ERROR: Server never wrote a response.
Seems to of worked perfectly, although listing in an empty directory obviously gives us no response in this case. We can quickly fix this by making an exception.
if length(command_split) == 1 || isnothing(f) || command_split[1] == "ls"
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if length(current_dir_files) == 0
write!(c, "no files found in this directory")
return
end
write!(c, join([filename for filename in current_dir_files], ";"))
return
end
Since we had to change so much to get this up and running, here is a look at our new, improved, and working module:
module ChiNAS
using Toolips
using TOML
using ReplMaker
using Toolips.Components
import Base: in, getindex
CONNECTED = "":0
abstract type AbstractRepository end
mutable struct Repository <: AbstractRepository
uri::String
file_count::Int64
end
mutable struct NASUser
ip::String
name::String
wd::String
end
mutable struct NASManager <: Toolips.AbstractExtension
hostname::String
home_dir::String
repos::Vector{AbstractRepository}
users::Vector{NASUser}
secret::String
end
mutable struct NASCommand{T <: Any} end
function in(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
~(isnothing(f))::Bool
end
function getindex(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
vec[f]::NASUser
end
logger = Toolips.Logger()
MANAGER = NASManager("", "", Vector{AbstractRepository}(), Vector{NASUser}(), "testpass")
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
path = replace(path, "\\" => "/")
MANAGER.home_dir = path * "/" * "home/"
dir_read::Vector{String} = readdir(path)
if ~("config.toml" in dir_read)
config_path::String = path * "/config.toml"
touch(config_path)
basic_dct = Dict("users" => Dict("admin" => Dict("ip" => "", "wd" => "~/")))
open(config_path, "w") do o::IO
TOML.print(o, basic_dct)
end
end
config = TOML.parse(read(path * "/config.toml", String))
MANAGER.users = [begin
info = config["users"][user]
NASUser(info["ip"], user, info["wd"])
end for user in keys(config["users"])]
start!(ChiNAS, ip:port)
secret::String = Toolips.gen_ref(5)
secret_path::String = path * "/secret.txt"
if ~("secret.txt" in dir_read)
touch(secret_path)
else
secret = read(secret_path, String)
end
open(path * "/secret.txt", "w") do o::IOStream
write(o, secret)
end
println("secret key: ", secret)
MANAGER.secret = secret
end
function connect(ip::String, port::Int64; path::String = pwd())
init_response::String = Toolips.get(ip:port)
if init_response == "secret?"
println("This storage requires a password to access.")
println("Please enter the access password:")
pwd = readline()
second_response = Toolips.get("http://$ip:$port/?secret=$pwd")
if second_response == "denied"
println("Access denied, it seems the password has been entered wrong.")
return(connect(ip, port, path = path))
end
println("Select a user-name from which to access this remote file-system:")
name = readline()
third_response = Toolips.get("http://$ip:$port/?name=$name")
if ~(third_response == "success")
println(third_response)
print("failure. name taken? for now we give up here.")
return
end
return(connect(ip, port, path = path))
end
global CONNECTED = ip:port
initrepl(send_to_connected,
prompt_text="$(ip):$(port) >",
prompt_color=:cyan,
start_key="-",
mode_name="Remote Filesystem")
end
function send_to_connected(line::String)
split_cmd::Vector{SubString} = split(line, " ")
if split_cmd[1] == "download"
download_url = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", "DOWNLOAD:$(split_cmd[2])")
return
end
response = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", replace(line, " " => ";"))
println(replace(response, ";" => "\n"))
end
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
client_ip::String = get_ip(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(client_ip, "", "~/")
push!(MANAGER.users, new_user)
write!(c, "success")
return
end
write!(c, "denied")
return
elseif :name in keys(args)
name::String = args[:name]
f = findfirst(user -> user.name == name, MANAGER.users)
if isnothing(f)
MANAGER.users[client_ip].name = name
else
MANAGER.users[f].ip = client_ip
end
write!(c, "success")
return
end
ips = [user.ip for user in MANAGER.users]
if ~(client_ip in ips)
write!(c, "secret?")
return
end
user = MANAGER.users[client_ip]
request = get_post(c)
command_split = split(request, ";")
f = findfirst(";", request)
if length(command_split) == 1 || isnothing(f) || command_split[1] == "ls"
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if length(current_dir_files) == 0
write!(c, "no files found in this directory")
return
end
write!(c, join([filename for filename in current_dir_files], ";"))
return
end
write!(c,
do_command(user, NASCommand{Symbol(command_split[1])}(),
command_split[2:length(command_split)] ...))
end
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
user.wd = join(wdsplit[1:length(wdsplit) - 1], "/")
end
end
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if ~(selected in current_dir_files)
return("ERROR: Directory does not exist to change into.")
end
user.wd = user.wd * selected * "/"
end
export main, default_404, logger, MANAGER
end
more commands
Now that we have a working server, we are going to add a few more commands. Fortunately, we created a our simple NASCommand parametric type from earlier, so implementing new commands is incredibly straightforward. We will start with making and removing directories and files.
mkdir
function do_command(user::NASUser, command::NASCommand{:mkdir}, args::SubString ...)
end
All of these are relatively straightforward, we are just going to use the Julia-equivalent command in that location.
function do_command(user::NASUser, command::NASCommand{:mkdir}, args::SubString ...)
mkdir(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("made directory: $(user.wd * args[1])")
end
rmdir
function do_command(user::NASUser, command::NASCommand{:rmdir}, args::SubString ...)
rmdir(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("removed directory: $(user.wd * args[1])")
end
rm
function do_command(user::NASUser, command::NASCommand{:rm}, args::SubString ...)
rm(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("removed file: $(user.wd * args[1])")
end
touch
function do_command(user::NASUser, command::NASCommand{:touch}, args::SubString ...)
touch(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("removed file: $(user.wd * args[1])")
end
Now let’s quickly test these commands; some of them might need to use the force key-word argument. So long as these all work we will then move to less straight-forward commands.
Select a user-name from which to access this remote file-system:
admin
REPL mode Remote Filesystem initialized. Press - to enter and backspace to exit.
"Prompt(\"127.0.0.1:8000 >\",...)"
127.0.0.1:8000 >ls
no files found in this directory
127.0.0.1:8000 >mkdir sample
made directory: ~/sample
127.0.0.1:8000 >ls
sample
127.0.0.1:8000 >cd sample
~/sample/
127.0.0.1:8000 >ls
no files found in this directory
127.0.0.1:8000 >mkdir sample2
made directory: ~/sample/sample2
127.0.0.1:8000 >ls
sample2
127.0.0.1:8000 >touch sample.txt
removed file: ~/sample/sample.txt
127.0.0.1:8000 >ls
sample.txt
sample2
127.0.0.1:8000 >rm sample.txt
removed file: ~/sample/sample.txt
127.0.0.1:8000 >ls
sample2
127.0.0.1:8000 >cd ..
ERROR: Directory does not exist to change into.
127.0.0.1:8000 >ls
sample2
I didn’t change the message in the touch function — also our cd .. did not work.
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
user.wd = join(wdsplit[1:length(wdsplit) - 1], "/")
end
end
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if ~(selected in current_dir_files)
return("ERROR: Directory does not exist to change into.")
end
user.wd = user.wd * selected * "/"
end
Looking back at our cd command, it is clear I wanted to return in the conditional up top. It is still puzzling though, as the working directory of course did not change. The cause of this is a little different, as our working directory always ends in / — by my own personal choice. We have to choose whether we want to add this slash each time we access this, or remove it each time we are in this situation. I chose to remove it, as the inverse makes file access easier. In this context, it creates a weird case where we actually have to index 2 splits below instead of 1, making sure to add a new / on the end to replace the one that disappeared as the third splits.
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
if length(wdsplit) > 2
user.wd = join(wdsplit[1:length(wdsplit) - 2], "/") * "/"
else
user.wd = "~/"
end
else
println(user.wd)
end
return(user.wd)::String
end
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if ~(selected in current_dir_files)
return("ERROR: Directory does not exist to change into.")
end
user.wd = user.wd * selected * "/"
user.wd::String
end
I also revised the other commands into perfection:
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
if length(wdsplit) > 2
user.wd = join(wdsplit[1:length(wdsplit) - 2], "/") * "/"
else
user.wd = "~/"
end
else
println(user.wd)
end
return(user.wd)::String
end
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if ~(selected in current_dir_files)
return("ERROR: Directory does not exist to change into.")
end
user.wd = user.wd * selected * "/"
user.wd::String
end
function do_command(user::NASUser, command::NASCommand{:mkdir}, args::SubString ...)
mkdir(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("made directory: $(user.wd * args[1])")
end
function do_command(user::NASUser, command::NASCommand{:rmdir}, args::SubString ...)
rmdir(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("removed directory: $(user.wd * args[1])")
end
function do_command(user::NASUser, command::NASCommand{:rm}, args::SubString ...)
rm(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("removed file: $(user.wd * args[1])")
end
function do_command(user::NASUser, command::NASCommand{:touch}, args::SubString ...)
touch(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("created file: $(user.wd * args[1])")
end
This is a good list of commands, but of course we are still missing something — download . This is of course essential for a NAS. The way we are going to handle downloads is interesting; client-side, we will send a request to download a file, then follow a link we receive to download the file. This extra step is mainly necessary because we need to download the file from a terminal — not just from a web-browser, so we cannot do it with GET requests. On the server-side, we need to find the file to download and generate a download route that will terminate itself once the download is done.
function do_command(user::NASUser, command::NASCommand{:download}, args::SubString ...)
route_path::String = "/" * Toolips.gen_ref(5)
new_r = route(route_path) do c::Connection
write!(c, read(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1], String))
f = findfirst(r -> r.path == route_path, c.routes)
deleteat!(c.routes, f)
end
push!(ChiNAS.routes, new_r)
return(route_path)::String
end
This simple command completes the server-side. As for the client, we need to modify our send_to_connected function:
function send_to_connected(line::String)
split_cmd::Vector{SubString} = split(line, " ")
if split_cmd[1] == "download"
download_url = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", "DOWNLOAD:$(split_cmd[2])")
return
end
response = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", replace(line, " " => ";"))
println(replace(response, ";" => "\n"))
end
We will first check if the length of splits is greater than 2. If it is, we will want to download to a specific path. If not, we will just go straight to download .
function send_to_connected(line::String)
split_cmd::Vector{SubString} = split(line, " ")
if split_cmd[1] == "download"
download_url = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", "download;$(split_cmd[2])")
path = pwd()
if length(split_cmd) > 2
path = split_cmd[3]
end
DLS.download("http://$(CONNECTED.ip):$(CONNECTED.port)" * download_url, path * "/$(split_cmd[2])")
return
end
response = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", replace(line, " " => ";"))
println(replace(response, ";" => "\n"))
end
DLS is the Module downloads, which is the return of the callable Base.Downloads . You might suspect this is a Module , Downloads is a Module — but Base.Downloads is a function. I suspect the reason this is the case is because the Downloads module requires has more dynamic requirements and maybe some initialization. So here is how you get DLS to call download :
Let’s hope this works!
julia> ChiNAS.connect("127.0.0.1", 8000)
This storage requires a password to access.
Please enter the access password:
dhjkt
Select a user-name from which to access this remote file-system:
admin
REPL mode Remote Filesystem initialized. Press - to enter and backspace to exit.
"Prompt(\"127.0.0.1:8000 >\",...)"
127.0.0.1:8000 >ls
sample
sample.txt
127.0.0.1:8000 >download sample.txt
127.0.0.1:8000 >
Let’s see if we have a new file!
shell> tree .
.
├── config.toml
├── home
│ ├── sample
│ │ └── sample2
│ └── sample.txt
├── repositories
├── sample.txt
└── secret.txt
5 directories, 4 files
Perfect! Essentially what we have done here is copied sample.txt through our server. And as expected, the route has disposed of itself:
julia> ChiNAS.routes
2-element Vector{Toolips.Route{Toolips.AbstractConnection}}:
404
/
404
404
/
/
Now that we have a basic version of this API now working and can perform basic tasks with files, I think it is finally time to start working on the real client. Here is a final look at our Julia module.
module ChiNAS
using Toolips
using TOML
using ReplMaker
using Toolips.Components
import Base: in, getindex
DLS = Base.Downloads()
CONNECTED = "":0
abstract type AbstractRepository end
mutable struct Repository <: AbstractRepository
uri::String
file_count::Int64
end
mutable struct NASUser
ip::String
name::String
wd::String
end
mutable struct NASManager <: Toolips.AbstractExtension
hostname::String
home_dir::String
repos::Vector{AbstractRepository}
users::Vector{NASUser}
secret::String
end
mutable struct NASCommand{T <: Any} end
function in(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
~(isnothing(f))::Bool
end
function getindex(vec::Vector{NASUser}, name::String)
f = findfirst(user -> user.ip == name, vec)
vec[f]::NASUser
end
logger = Toolips.Logger()
MANAGER = NASManager("", "", Vector{AbstractRepository}(), Vector{NASUser}(), "testpass")
function host(ip::String, port::Int64; path::String = pwd(), hostname::String = "chiNAS")
path = replace(path, "\\" => "/")
MANAGER.home_dir = path * "/" * "home/"
dir_read::Vector{String} = readdir(path)
if ~("config.toml" in dir_read)
config_path::String = path * "/config.toml"
touch(config_path)
basic_dct = Dict("users" => Dict("admin" => Dict("ip" => "", "wd" => "~/")))
open(config_path, "w") do o::IO
TOML.print(o, basic_dct)
end
end
config = TOML.parse(read(path * "/config.toml", String))
MANAGER.users = [begin
info = config["users"][user]
NASUser(info["ip"], user, info["wd"])
end for user in keys(config["users"])]
start!(ChiNAS, ip:port)
secret::String = Toolips.gen_ref(5)
secret_path::String = path * "/secret.txt"
if ~("secret.txt" in dir_read)
touch(secret_path)
else
secret = read(secret_path, String)
end
open(path * "/secret.txt", "w") do o::IOStream
write(o, secret)
end
println("secret key: ", secret)
MANAGER.secret = secret
end
function connect(ip::String, port::Int64; path::String = pwd())
init_response::String = Toolips.get(ip:port)
if init_response == "secret?"
println("This storage requires a password to access.")
println("Please enter the access password:")
pwd = readline()
second_response = Toolips.get("http://$ip:$port/?secret=$pwd")
if second_response == "denied"
println("Access denied, it seems the password has been entered wrong.")
return(connect(ip, port, path = path))
end
println("Select a user-name from which to access this remote file-system:")
name = readline()
third_response = Toolips.get("http://$ip:$port/?name=$name")
if ~(third_response == "success")
println(third_response)
print("failure. name taken? for now we give up here.")
return
end
return(connect(ip, port, path = path))
end
global CONNECTED = ip:port
initrepl(send_to_connected,
prompt_text="$(ip):$(port) >",
prompt_color=:cyan,
start_key="-",
mode_name="Remote Filesystem")
end
function send_to_connected(line::String)
split_cmd::Vector{SubString} = split(line, " ")
if split_cmd[1] == "download"
download_url = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", "download;$(split_cmd[2])")
path = pwd()
if length(split_cmd) > 2
path = split_cmd[3]
end
DLS.download("http://$(CONNECTED.ip):$(CONNECTED.port)" * download_url, path * "/$(split_cmd[2])")
return
end
response = Toolips.post("http://$(CONNECTED.ip):$(CONNECTED.port)", replace(line, " " => ";"))
println(replace(response, ";" => "\n"))
end
main = route("/") do c::Toolips.AbstractConnection
args = get_args(c)
client_ip::String = get_ip(c)
if :secret in keys(args)
if args[:secret] == MANAGER.secret
new_user = NASUser(client_ip, "", "~/")
push!(MANAGER.users, new_user)
write!(c, "success")
return
end
write!(c, "denied")
return
elseif :name in keys(args)
name::String = args[:name]
f = findfirst(user -> user.name == name, MANAGER.users)
if isnothing(f)
MANAGER.users[client_ip].name = name
else
MANAGER.users[f].ip = client_ip
end
write!(c, "success")
return
end
ips = [user.ip for user in MANAGER.users]
if ~(client_ip in ips)
write!(c, "secret?")
return
end
user = MANAGER.users[client_ip]
request = get_post(c)
command_split = split(request, ";")
f = findfirst(";", request)
if length(command_split) == 1 || isnothing(f) || command_split[1] == "ls"
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if length(current_dir_files) == 0
write!(c, "no files found in this directory")
return
end
write!(c, join([filename for filename in current_dir_files], ";"))
return
end
write!(c,
do_command(user, NASCommand{Symbol(command_split[1])}(),
command_split[2:length(command_split)] ...))
end
function do_command(user::NASUser, command::NASCommand{:cd}, args::SubString ...)
selected::String = string(args[1])
if selected == ".."
if user.wd != "~/"
wdsplit = split(user.wd, "/")
if length(wdsplit) > 2
user.wd = join(wdsplit[1:length(wdsplit) - 2], "/") * "/"
else
user.wd = "~/"
end
else
println(user.wd)
end
return(user.wd)::String
end
current_dir_files = readdir(replace(user.wd, "~/" => MANAGER.home_dir * "/"))
if ~(selected in current_dir_files)
return("ERROR: Directory does not exist to change into.")
end
user.wd = user.wd * selected * "/"
user.wd::String
end
function do_command(user::NASUser, command::NASCommand{:mkdir}, args::SubString ...)
mkdir(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("made directory: $(user.wd * args[1])")
end
function do_command(user::NASUser, command::NASCommand{:rmdir}, args::SubString ...)
rmdir(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("removed directory: $(user.wd * args[1])")
end
function do_command(user::NASUser, command::NASCommand{:rm}, args::SubString ...)
rm(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("removed file: $(user.wd * args[1])")
end
function do_command(user::NASUser, command::NASCommand{:touch}, args::SubString ...)
touch(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1])
return("created file: $(user.wd * args[1])")
end
function do_command(user::NASUser, command::NASCommand{:download}, args::SubString ...)
route_path::String = "/" * Toolips.gen_ref(5)
new_r = route(route_path) do c::AbstractConnection
write!(c, read(replace(user.wd, "~/" => MANAGER.home_dir * "/") * args[1], String))
f = findfirst(r -> r.path == route_path, c.routes)
deleteat!(c.routes, f)
end
push!(ChiNAS.routes, new_r)
return(route_path)::String
end
export main, default_404, logger, MANAGER
end
Client Application
Now that we have a working API in the form of our Julia-bourne NAS server, we now need to create a client-side application that will utilize the API and create a more streamlined, user-based experience. The reason I am developing this client is so that people on my network who are not developers and use other operating systems will still be able to access the files. While the CLI works fine, each user that wants to use the CLI will need to install Julia and then use the package before calling connect — not only is that a lot of work but it is above a lot of people’s pay-grade.
I recently wrote an article about Godot, why I think it is great, and why I think more than just game developers should look into what it has to offer:
In this article, I go into a lot of detail on developing applications in Godot and why it is a great “ framework” to work with — not only for games, but also applications. We are able to work in many different languages, compile for many different platforms, and it is relatively fast and easy to get a UI up and running. Considering this, and how cool it would be to make an application that uses both, I have decided to use Godot to build the client for this project. I will start by making a new project in my Godot engine. I will also modify the resolution (to 1080p) and set the stretch mode of my project to viewport.
app source
authentication
We will start with the secret and username handshake that we performed prior with the other client. First, we will create a simple UI for entering the server to connect to. I will start by making us a background and a window to add UI elements to. Both of these come in the form of a ColorRect , which are scaled. The connect_page rect has a VBoxcontainer as a child, which will hold all of the components of our individual form. The ColorRect that parents this is anchored to the center.
Adding the elements of our form, I am happy with the result:
Now we can move onto scripting this thing. We will start by attaching a script to main .
extends Control
func _ready():
pass
func _process(delta):
pass
We will bind the signal of the connect button being pressed to a new function.
When this button is pressed, we will go ahead and make a request to the server — ensuring the server is on and present before moving on to verifying the secret. This follows the same procedure as the client we created in Julia, and the reason I am still choosing this here is so that we can provide feedback on if the key is bad or the server is bad. I have never done this in Godot, so I decided to start by looking at Godot’s documentation on making HTTP requests. Here is the page:
extends Node
func _ready():
$HTTPRequest.request_completed.connect(_on_request_completed)
$HTTPRequest.request("https://api.github.com/repos/godotengine/godot/releases/latest")
func _on_request_completed(result, response_code, headers, body):
var json = JSON.parse_string(body.get_string_from_utf8())
print(json["name"])
From their example, I have a pretty good idea of how this works. To post, we simply provide more arguments to request .
var headers = ["Content-Type: application/json"]
$HTTPRequest.request(url, headers, HTTPClient.METHOD_POST, json)
I am just going to create my request as a member variable instead of adding it to the scene tree.
extends Control
var HTTP = HTTPRequest.new()
func _ready():
pass
func _process(delta):
pass
func _on_connect_button_pressed():
pass
In the _on_connect_button_pressed function, we can now make a request to the server and do our same secret handshake from before. I will start by connecting this signal and then making the request in _on_connect_button_pressed .
extends Control
var HTTP : HTTPRequest = HTTPRequest.new()
var connected_ip : String = ""
var connected_port : int = 0
var connecting = false
func _ready():
pass
func _process(delta):
pass
func _on_connect_button_pressed():
connecting = true
connected_ip = $background/connect_page/connect_container/HboxContainer/ipbox.text
connected_port = $background/connect_page/connect_container/HboxContainer/portbox.value
HTTP.request_completed.connect(on_init_response)
HTTP.request("http://" + connected_ip + ":" + str(connected_port))
func on_init_response(result, response_code, headers, body):
pass
Now will try to provide the secret back to the server in response.
func on_init_response(result, response_code, headers, body):
HTTP.request_completed.disconnect(on_init_response)
HTTP.request_completed.connect(on_secret_response)
var secret = $background/connect_page/connect_container/secretbox.text
HTTP.request("http://" + connected_ip + ":" + str(connected_port) + "/?secret=" + secret)
On the initial response, we rebind the request_completed signal to on_secret_response . In that function, we finish by asking the user for a name. For now, I will just ensure we receive the authentication success message. The body passed as an argument to these functions is actually a PackedByteArray , not a String , so we will decode it using its member method get_string_from_utf8 .
func on_secret_response(result, response_code, headers, body):
body = body.get_string_from_utf8()
if body != "denied":
print(body)
print("we were successful lol")
HTTP.request_completed.disconnect(on_secret_response)
That worked as expected. Let’s go ahead and make the name prompt.
We will make the connect_page invisible and make the username box visible when we confirm the secret. We will then bind our name set request to this login button.
func on_secret_response(result, response_code, headers, body):
body = body.get_string_from_utf8()
if body != "denied":
$background/connect_page.visible = false
$background/username_box.visible = true
HTTP.request_completed.disconnect(on_secret_response)
I will tie a new signal from the press of the login button to a new function where we will send that data in a GET argument. After this, it is finally time to start representing the actual files remotely.
func _on_login_button_pressed():
var name = $background/username_box/VBoxContainer/usernamebox.text
HTTP.request_completed.connect(on_nameset)
HTTP.request("http://" + connected_ip + ":" + str(connected_port) + "/?name=" + name)
func on_nameset(result, response_code, headers, body):
var url = "http://" + connected_ip + ":" + str(connected_port)
$username_box.visible = false
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "ls")
At the end of this, in on_nameset we will go ahead and make a post request and get rid of the username_box . Let’s bind this to one last function, we will print the body we retrieve in return and ensure it is file/directory names separated by semi-colons.
func on_nameset(result, response_code, headers, body):
var url = "http://" + connected_ip + ":" + str(connected_port)
$background/username_box.visible = false
HTTP.request_completed.disconnect(on_nameset)
HTTP.request_completed.connect(recieve_ls)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "ls")
func recieve_ls(result, response_code, headers, body):
body = body.get_string_from_utf8()
print(body)
After logging in again, the screen went blank and I got the expected output in the terminal.
file ui
Now that we have the request-response system working and we can receive files from the storage, it is time to make a browse-able file-system that we can use to navigate the remote file-system from this application. I have decided to use the ItemList for this.
My plan is to have the left side be reserved for broader file navigation — the repositories, files, current directory. The center will hold the current directory we are looking at, and the right will hold control for the selected file. With this, we simply set the elements of the ItemList each time the ls command is called.
func recieve_ls(result, response_code, headers, body):
body = body.get_string_from_utf8()
var splits = body.split(";")
for file_dir in splits:
pass
For now, I have limited the complexity by only sending the names — not a lot of information about the file. While I intend to eventually change this, for now we will determine if a name is a file or directory by checking if the name contains a . . In the future, we will send more file information in each split, including whether or not that file is a directory.
func on_nameset(result, response_code, headers, body):
var url = "http://" + connected_ip + ":" + str(connected_port)
$background/username_box.visible = false
HTTP.request_completed.disconnect(on_nameset)
HTTP.request_completed.connect(recieve_ls)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "ls")
$background/main_page.visible = true
func recieve_ls(result, response_code, headers, body):
body = body.get_string_from_utf8()
var splits = body.split(";")
items.clear()
items.add_item("..")
for file_dir in splits:
items.add_item(file_dir)
This already gives us each file listed out:
We will also need to keep track of which file or directory is at each index. In the future, we may again use more complex data for this — for now, we will just store the names in an organized list that corresponds to the order we added the items.
extends Control
@onready var HTTP : HTTPRequest = $HTTPRequest
@onready var items : ItemList = $background/main_page/files_panel/ItemList
var selected_files = []
var connected_ip : String = ""
var connected_port : int = 0
var connecting = false
func receive_ls(result, response_code, headers, body):
body = body.get_string_from_utf8()
var splits = body.split(";")
items.clear()
items.add_item("..")
for file_dir in splits:
selected_files.append(file_dir)
items.add_item(file_dir)
Now we will just bind signals from the item list to new commands.
func _on_item_list_item_activated(index):
pass
func _on_item_list_item_selected(index):
pass
on_em_list_item_activated will be called when an item is either double-clicked or selected and Enter is pressed. on_item_list_item_selected will be called by simply clicking once. The former will be used for navigation, whereas the latter will be used to view more information. We will start by changing the directory to the selected index.
func _on_item_list_item_activated(index):
var selected_file = selected_files[index]
if not selected_file.contains("."):
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "cd;" + selected_file)
I made a change in making url a member variable that updates, as we probably want to reuse this. After that cd post is completed, we will post again with ls , recycling the function from earlier.
func receive_cd(result, response_code, headers, body):
HTTP.request_completed.connect(recieve_ls)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "ls")
func receive_ls(result, response_code, headers, body):
body = body.get_string_from_utf8()
var splits = body.split(";")
items.clear()
items.add_item("..")
for file_dir in splits:
selected_files.append(file_dir)
items.add_item(file_dir)
HTTP.request_completed.disconnect(receive_ls)
func receive_cd(result, response_code, headers, body):
HTTP.request_completed.connect(receive_ls)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "ls")
func _on_item_list_item_activated(index):
var selected_file = selected_files[index]
if not selected_file.contains("."):
HTTP.request_completed.connect(receive_cd)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "cd;" + selected_file)
func _on_item_list_item_selected(index):
pass # Replace with function body.
In the LS function, I am also now gouing to clear our selected_files list.
func receive_ls(result, response_code, headers, body):
body = body.get_string_from_utf8()
var splits = body.split(";")
items.clear()
items.add_item("..")
selected_files = [".."]
for file_dir in splits:
selected_files.append(file_dir)
items.add_item(file_dir)
HTTP.request_completed.disconnect(receive_ls)
This worked for the most part — only problem is I wasn’t able to move up directories… The culprit, rather hilariously, is that we excluded anything with a . from being a directory.
func _on_item_list_item_activated(index):
var selected_file = selected_files[index]
if not selected_file.contains(".") or selected_file == "..":
HTTP.request_completed.connect(receive_cd)
print(selected_file)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "cd;" + selected_file)
This worked! Now we we are going finish off this initial version by adding downloading for individual files.
func _on_item_list_item_selected(index):
pass
I will update this aspect of the UI when we have more data to present — for now we will focus on selecting and downloading the selected file.
func _on_item_list_item_selected(index):
var selected_file = selected_files[index]
if not selected_file.contains("."):
return
active_file = index
var file_info = $background/main_page/inspector_panel/fileinfo
file_info.visible = true
file_info.get_child(0).text = selected_file
file_info.get_child(1).text = selected_file.split(".")[2] + " file"
I added a new member variable, active_file to track the currently selected file. We will now use this in a new function that will be called when we click download.
func _on_download_selected_pressed():
pass
func _on_download_selected_pressed():
HTTP.request_completed.connect(receive_download)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "download;" + selected_files[active_file])
While I was going to make a download request here, we likely want to first pick a file to download into. For this, we will create a new FileDialog .
func _on_download_selected_pressed():
$download_popup.popup()
func _on_download_popup_confirmed():
var popup = $download_popup
print(popup.current_path + "/" + popup.current_file)
# HTTP.request_completed.connect(receive_download)
# HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "download;" + selected_files[active_file])
This is getting pretty close, I just want to print the current file path and file name in order to ensure everything is correct. Trying this for the first time, I found out we actually want to use the item_clicked signal, not the one I used — item_selected . In that function I also indexed the file split at 2 — please cut me some slack, we were just in Julia where the indexes start at one.
func _on_item_list_item_selected(index, pos, mb):
var selected_file = selected_files[index]
if not selected_file.contains("."):
return
active_file = index
var file_info = $background/main_page/inspector_panel/fileinfo
file_info.visible = true
file_info.get_child(0).text = selected_file
file_info.get_child(1).text = selected_file.split(".")[1] + " file"
Now we can select files, so let’s try downloading them.
So we were close, turns out the current directory comes with the file name attached.
func _on_download_popup_confirmed():
var popup = $download_popup
dl_path = popup.current_path
HTTP.request_completed.connect(receive_download)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "download;" + selected_files[active_file])
func receive_download(result, response_code, headers, body):
HTTP.request_completed.disconnect(receive_download)
# download URL
body = body.get_string_from_utf8()
# we set download file, and make GET request to download link.
HTTP.download_file = dl_path
HTTP.request_completed.connect(finalize_download)
HTTP.request(url + body)
This time, we set the download_file and make a GET request. We are going to have to find out if is just works — I have never downloaded files like this in Godot so we will soon see.
I again downloaded our sample.txt — les try opening the file.
Absolute perfection!
Here is one final look at our script for this app:
extends Control
@onready var HTTP : HTTPRequest = $HTTPRequest
@onready var items : ItemList = $background/main_page/files_panel/ItemList
var selected_files = []
var active_file : int = 0
var connected_ip : String = ""
var connected_port : int = 0
var connecting = false
var url : String
var dl_path : String
# Called when the node enters the scene tree for the first time.
func _ready():
pass # Replace with function body.
# Called every frame. 'delta' is the elapsed time since the previous frame.
func _process(delta):
pass
func _on_connect_button_pressed():
connecting = true
connected_ip = $background/connect_page/connect_container/HBoxContainer/ipbox.text
connected_port = $background/connect_page/connect_container/HBoxContainer/portbox.value
HTTP.request_completed.connect(on_init_response)
HTTP.request("http://" + connected_ip + ":" + str(connected_port))
# the server is up, we provide the key
func on_init_response(result, response_code, headers, body):
HTTP.request_completed.disconnect(on_init_response)
HTTP.request_completed.connect(on_secret_response)
var secret = $background/connect_page/connect_container/secretbox.text
HTTP.request("http://" + connected_ip + ":" + str(connected_port) + "/?secret=" + secret)
# secret verified, we ask for username
func on_secret_response(result, response_code, headers, body):
body = body.get_string_from_utf8()
if body != "denied":
$background/connect_page.visible = false
$background/username_box.visible = true
HTTP.request_completed.disconnect(on_secret_response)
func _on_login_button_pressed():
var name = $background/username_box/VBoxContainer/usernamebox.text
HTTP.request_completed.connect(on_nameset)
HTTP.request("http://" + connected_ip + ":" + str(connected_port) + "/?name=" + name)
url = "http://" + connected_ip + ":" + str(connected_port)
func on_nameset(result, response_code, headers, body):
$background/username_box.visible = false
HTTP.request_completed.disconnect(on_nameset)
HTTP.request_completed.connect(receive_ls)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "ls")
$background/main_page.visible = true
func receive_ls(result, response_code, headers, body):
body = body.get_string_from_utf8()
var splits = body.split(";")
items.clear()
items.add_item("..")
selected_files = [".."]
for file_dir in splits:
selected_files.append(file_dir)
items.add_item(file_dir)
HTTP.request_completed.disconnect(receive_ls)
func receive_cd(result, response_code, headers, body):
HTTP.request_completed.disconnect(receive_cd)
HTTP.request_completed.connect(receive_ls)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "ls")
func receive_download(result, response_code, headers, body):
HTTP.request_completed.disconnect(receive_download)
# download URL
body = body.get_string_from_utf8()
# we set download file, and make GET request to download link.
HTTP.download_file = dl_path
HTTP.request_completed.connect(finalize_download)
HTTP.request(url + body)
func finalize_download(result, response_code, headers, body):
pass # for now this will do nothing, in the future it will present a message.
func _on_item_list_item_activated(index):
var selected_file = selected_files[index]
if not selected_file.contains(".") or selected_file == "..":
HTTP.request_completed.connect(receive_cd)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "cd;" + selected_file)
func _on_item_list_item_selected(index, pos, mb):
var selected_file = selected_files[index]
if not selected_file.contains("."):
return
active_file = index
var file_info = $background/main_page/inspector_panel/fileinfo
file_info.visible = true
file_info.get_child(0).text = selected_file
file_info.get_child(1).text = selected_file.split(".")[1] + " file"
func _on_download_selected_pressed():
$download_popup.popup()
func _on_download_popup_confirmed():
var popup = $download_popup
dl_path = popup.current_path
HTTP.request_completed.connect(receive_download)
HTTP.request(url, ["Content-Type: text"], HTTPClient.METHOD_POST, "download;" + selected_files[active_file])
conclusion
Combining the capabilities of local software and server software can be incredibly powerful! I have always wanted to have network-attached storage, and making my own software presents a host of advantages to me personally. It was really exciting to be able to use such a diverse tech-stack to bring that into fruition. While Julia is a great programming language, it does have short-comings like any language and the among most significant of those are likely the lack of compiled applications, somewhat high memory usage (with few ways to manually manage the memory usage,) and requiring the language’s executable to run. Godot has its own limitations as well in regards to building software, but seems to excel in being easy to port to many platforms. With Godot, this application could easily become a mobile app that gives me access to my files from my phone.
While this rather basic version isn’t perfect, it does already accomplish a lot of the routines and tasks we might typically associate with network-attached storage. There are issues, both found and yet to be found. With a bit more work, in a future article we will wrap this into a more usable experience. In fact, we will likely have another article of development and an article on the deployment of this project on my local network. I fully intend to share everything one would need to know to create this themselves. Next time we work on this project, we will be adding…
- A more detailed system for relaying information on files
- more commands
- repositories
- a file tree
- … and I am certain a lot more.
In a lot of ways, this is a great proof of concept. While Toolips is a great tool for building websites, it is also a great tool for building APIs. It is easy to see why creating a front-end in Godot like this and a back-end in Julia has some advantages, especially for web-integrated projects like this. Though this article is now a 40 minute read (sorry,) I am happy with all that we managed to accomplish. Consider that a Minimum Viable Product of this type would usually take a lot longer; thanks to clever programming strategy and the amazing tools we have at our disposal in the open-source community in 2024, we can easily create what we want for both experience and pleasure. Today that turned out to be an awesome application to handle my Network-Attached-Storage!