SecureSight Data Stream

Secure Real-Time Data Visualization Platform

The digital age we inhabit sees an ever-growing dependency on data, thereby amplifying the urgency of data security. Journey with me through this compelling project where I amalgamate the robustness of CryptoJS encryption with a NoSQL database, specifically MongoDB, to encrypt and securely store JSON data.

1. Database Creation and Structuring

The foundation of the project is a robust MySQL database, structured to store and manage handwritten images of Chinese Pinyin. Within the database, a dedicated table is provisioned, where each record consists of a unique identifier (like table_id) and its corresponding image data, stored in BLOB format. Efficiently indexed and organized, this setup ensures that images can be swiftly retrieved and processed, laying the groundwork for the successive machine learning pipeline.

One more point is since my goal is to be able to upload/ download data files from the database and encode/ decode the file as one continuous process, I also need to make the program streamline the process of getting data, encrypting it, and storing it securely. It allows the user to continuously feed data and get instant feedback, enhancing user experience and efficiency.

# Main loop for real-time data input
while True:
    try:
        user_input = input("Please input your JSON data (or type 'exit' to quit): ")
        if user_input.lower() == 'exit':
            break

        # Checking if the input is valid JSON
        try:
            json.loads(user_input)
            encrypted_data = encrypt_data(user_input, key)
            upload_to_mongo(encrypted_data)
            print("Data encrypted and uploaded to MongoDB successfully!")

        except json.JSONDecodeError:
            print("Invalid JSON. Please ensure your input is a valid JSON format.")

2. JSON Data Encryption

With Python’s PyMongo library, we source the data destined for encryption. The omnipotent CryptoJS library then steps in to encrypt each fragment of JSON data prior to its archival. The nested nature of JSON mandates a profound traversal and encryption of every embedded object or array, ensuring every last bit of data is securely encrypted. This guarantees that even in the unfortunate event of unauthorized database access, the data remains inscrutable.

# Encrypt data with AES-CBC and return ciphertext and HMAC
# Import Crypto Modules...
def encrypt_data(data, key):
    hmac_key = key[:16]  # First half of the key for HMAC
    enc_key = key[16:]  # Second half of the key for encryption
    
    iv = get_random_bytes(16)
    cipher = AES.new(enc_key, AES.MODE_CBC, iv)
    encrypted = cipher.encrypt(pad(data.encode(), 16))
    
    # Calculate HMAC for encrypted data and append it
    hmac = HMAC.new(hmac_key, encrypted, digestmod=SHA256).digest()
    return base64.b64encode(iv + encrypted + hmac)

The encryption process employs the Advanced Encryption Standard (AES) in Cipher Block Chaining (CBC) mode, combined with HMAC for data integrity. Initially, a password-derived symmetric key is split into encryption and HMAC subkeys. Input data, typically in JSON format, is encrypted using the AES subkey and an initialization vector (IV). After encryption, HMAC is applied to the encrypted data to generate an integrity code. The IV, encrypted data, and HMAC are concatenated and encoded using base64, producing the final encrypted output. This approach ensures both data confidentiality through AES and data integrity via HMAC, delivering robust and secure encryption.

3. Data Visualization and Utilization

To make the data fetching process smoother, I also incorporated an intuitive data visualization mechanism to instantly translate decrypted JSON content into neat tabular forms. Leveraging the power of pandas, nested JSON data is seamlessly segmented and presented as distinct, clearly labeled tables. This not only bolsters comprehension by offering a structured view but also facilitates immediate data analysis. By transforming intricate nested structures into easily digestible tables, we’re ensuring that even as data complexity grows, its accessibility and clarity remain uncompromised.

decrypted_data = decrypt_data(encrypted_data, key)
    json_data = json.loads(decrypted_data)
    
    tables = json_to_table(json_data)
    
    for table_name, table_data in tables.items():
        print(f"Table: {table_name}")
        display(table_data)

4. Results and Applications

This project seamlessly blends robust AES encryption with intuitive data visualization. This ensures data remains secure, while also being easily accessible in a clear, tabular format. By merging security with user-friendliness, we’ve established a system that safeguards information without sacrificing clarity.

Beyond database management, the methods used in this project also have vast potential in fields like finance and healthcare, where data security is crucial. The encryption safeguards sensitive details, and the visualization aids in quick decision-making.

Please feel free to contact me if you would like to learn more about this exciting project or if you have any inquiries related to my skills and experience.