Intro

There are so many fascinating projects out there, especially in the realm of embedded databases. Projects like Redb, LMDB, and Fjäll each offer impressive features such as thread safety, ACID compliance, partitioning, and more. These are all fantastic innovations, and they certainly serve their purpose well. However, I’m a simple person. I only want dumb simple things until I need more complex things then I need complex things.

With that said, let’s build a simple JSON database in Rust!

Why

I’m building a Postman equivalent called blink, but blink has more of a focus on orchestrating API calls rather than managing individual APIs or tags like you see in Insomnia, Postman Bruno etc… It’s more like if Postman Flows were front and center, but I digress.

Now just like other application users will be able to close the blink and if they are allowed to do that, then we need to persist the various API calls and the end-to-end orchestration logic to a file system. I decided to go with JSON. XML might’ve been a good option too, but I’m not a fan of XML—and hey, it’s my project, so… JSON it is!

Requirements

So what do I need? Let’s check

  • Read many json files
  • Read values from json files
  • Update values
  • Persist values into json files again

Design

This isn’t rocket science—we mainly need serde_json for serialization and deserialization, along with a simple API. Performance and error handling isn’t a consideration.

Api Design

load_files(path) -> Database load all the files in this directory get_all(&self) -> Vec<&T> update_by_id(id: &str, T val) -> bool Update the json with some new values for now persist to the file automatically.

That’s it we don’t need anything more for now.

Let’s Start!

Generate The Lib

cargo new dumb-json-db --lib

It’s a great name cause it is what the name is.

The easy part

Add serde and serde_json to the project

cargo add serde
cargo add serde_json

And we need some way to identify the json documents. Uuid seems like a good choice for now so….

cargo add uuid -F std,v4

The not so easy part

Let’s build out that API. For now let’s assume there will only be 1 JSON schema that works with blink.

extern crate serde_json;
extern crate uuid; 

use uuid::Uuid;
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use std::fs; 

// id is the primary key so we need to be able to get that value no matter what
pub trait Identifyable {
    fn get_id(&self) -> &Uuid;
}

// Meta is an object we keep the data and the path to the json file so that 
// we can easily write to the file that is associated to the file
struct Meta<T> where T: Identifyable + for <'a> Deserialize<'a> + Serialize + Sized {
    pub value: T, 
    pub file_path: PathBuf
}

// Contains our database value etc... 
pub struct  Database<T> where T: Identifyable + for <'a> Deserialize<'a> + Serialize + Sized   {
    values: Vec<Meta<T>>
}


impl <T> Database<T> where T: Identifyable + for <'a> Deserialize<'a> + Serialize + Sized {
    
    fn load_dir(path_to_dir: PathBuf) -> Database<T> {
        let mut database: Vec<Meta<T>> = Vec::new();
        // we assume there is a "folder" that we read in our json files 
        if path_to_dir.is_dir() {
            // for each file read in json and add it to the database
            for entry in fs::read_dir(path_to_dir).unwrap() {
                let entry = entry.unwrap();
                let file_path = entry.path();
                let contents = fs::read_to_string(&file_path).expect("Unable to read the database files!!!!");
                let val: T  = serde_json::from_str(&contents).expect("Invalid json scema maybe it got corrupted?");
                let meta : Meta<T> = Meta {
                    value: val,
                    file_path: file_path 
                };
                database.push(meta); 
            }
            return Database {
                values: database
            };
        }
        return Database {
            values: database
        };
    }

    pub fn get_all(&self) -> Vec<&T>  {
        // return all the database value without the file path
        self.values.iter().map(|val| &val.value).collect()
    }

    // updates the database entry based on primary key and persists it to the file
    pub fn update_by_id(&mut self, val: T) -> bool {
        
        let position = self.values.iter().position(|value| value.value.get_id() == val.get_id()); 
        if let Some(pos) = position {
            let json = serde_json::to_string(&val).expect("Failed to translate to JSON?"); 
            fs::write(&self.values[pos].file_path, json).expect("Failed to persist to database file system!"); 
            self.values[pos].value = val; 
            true
        } else {
            false 
        }
    }

}

It’s horribly simple with little no to error handling. In the future maybe we need some error handling and some other fancy features, but this is good enough.

Next up

Let’s integrate this into blink!