Version 0.3.1

Actually working properly!

CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog from 0.2.0 to 0.3.0
+
+## Major changes
+
+* Improved multithreading engine to be actually multithreaded
+
+* Gave the Python extension a beter name
+
+* Getting ready for feature-freeze.
+
+## Minor changes
+
+* Fixed bugs pertaining to proxy
+
+* Attempted fix at hanging socket by introducing a default 25 second timeout.
+
+## Configuration changes
+
+* Added a new `threading` key to define if threading should be enabled.
+
+

PES.md

@@ -0,0 +1,43 @@
+# PES
+
+This is a quick reference document on how to implement PES.
+
+## Example script:
+
+The default script is the following:
+
+```python
+import sys
+import os
+
+if not os.getcwd() in sys.path:
+    sys.path.append(os.getcwd())
+import amethyst
+
+
+class PES:
+    def __init__(self):
+        # DO NOT USE THIS FUNCTION FOR PROGRAM, ONLY ON_REQUEST PLEASE!!
+        # WARNING: ONLY CHANGE THE THREAD_SAFETY VARIABLE! DO NOT ADD OR REMOVE ANYTHING!
+        # THEY WILL COMPROMISE ANY THREAD-SAFETY SECURITY MECHANISMS AMETHYST HAS IN PLACE!
+        # YOU HAVE BEEN WARNED!
+        self.build_response = amethyst.WebServer.build_binary_response
+        self.fh = amethyst.FileHandler("..")
+        self.rq = amethyst.RequestParser()
+        self.THREAD_SAFETY: bool = True
+
+    def on_request(self, req):
+        return self.build_response(200, b"Heyhey! This is the default PES script!", "text/html")
+```
+
+## Threading and Thread-safety.
+
+When you execute PES scripts, they have a high chance of not being thread-safe because they edit the page you visit. It means you cannot guarantee the data you want is actually the data getting sent over the wire. This issue can be fixed in multiple ways:
+
+1. Disabling threading, then only one script can run at once, but this costs a lot of performance
+
+2. Enforcing thread-safety on all scripts, this will mean every script can be trusted, but makes development difficult (especially for people with no coding experience)
+
+3. Implementing mechanisms that will try to patch up holes if threading is enabled and a thread-unsafe script is loaded.
+
+Amethyst uses fix 3. It only allows one script to run at a time (if both threading is enabled and a thread-unsafe script is loaded), but if you specify only one host to use PES, all other hosts still enjoy the benefits of a multithreaded server! This makes sure that if user 1 and user 2 both request something and PES is involved, user 1 receives part of the data they want and part of the data user 2 wants and vice versa. This security mechanism only works if you respect them. Amethyst will throw a warning if you have a situation you have an unsafe script and run threaded, this warning isn't critical, as the script will still execute with security mechanisms, but the warning in the script is clear. Amethyst can't help if you make it explicitly unsafe yourself.

README.md

@@ -1,32 +1,115 @@
 # Amethyst Web Server
 
 ## A word of warning!
+
 Currently Amethyst is in very early alpha stage, a lot of things will be broken, names won't be correct,
 promised features missing, but I'm very much working on it live!  
-Every save I do increments the build number by 1, I won't publish all of them, but most of them will be published.  
+Every save I do increments the build number by 1, I won't publish all of them, but some of them will be published.  
 Once a milestone is hit (e.g. a new feature fully implemented), I'll publish a release!
 
+## Approaching 1.0.0!
+
+Amethyst is finally approaching 1.0.0! Very very soon I will feature-freeze the project and begin just fixing bugs and cleaning up code! This may take a bit because the codebase is very cluttered, and because all features are there in a basic state, it would be better to fix and clean up what I have, so I have a workable codebase for implementing new features, and because new features aren't going to be added anyway, I might as well fully release the project!
+
 ## Currently working features:
-* New configuration is ~75% done, most features work.
+
+* New configuration is ~95% done, most features work.
 * Fixed **A LOT** of unreported bugs from the old code.
 * More resilliency against errors.
 * Improved security.
+* Proxy almost working!
 
 ## Project status:
-Amethyst will stay in beta for a while, I want all features to work, put I will make pre-release versions that are mostly stable.  
+
+Amethyst will stay in beta for a while, I want all features to work, but I will make pre-release versions that are mostly stable.  
 They can be found as the `amethyst-prerel-0.a.b` releases. I won't guarantee 100% stability, but waay more than just some random build.  
 
 ## Install instructions:
-Install Python, and change the provided config.
+
+Install Python, execute `amethyst.py` and change the provided config.
 
 ## Minimum requirements:
-Python 3.8+  
+
+Python 3.10+  
 And whatever PC that happens to run that.  
-I recommend Python 3.10 or above though, with a PC running:
+I recommend Python 3.12 or above though, with a PC running:
+
 * Windows 8.1+
 * macOS 10.15+
 * Linux 4.19+
 * FreeBSD 13.2R+
 * Some other somewhat recent OS.
 
-## Currently W.I.P. Check back later!
+## The webserver itself:
+
+The Amethyst webserver is meant to be easy to use and configure. Its configuration takes inspiration from nginx and Caddyfile.  
+The language the configuration is made in is AmethystConf.
+The default config is as follows:
+
+```amethystconf
+host * {
+    directory:./html
+    pesmode:0
+    index:index.html
+}
+
+globals {
+    http:1
+    https:1
+    port:8080
+    https-port:8443
+    key:./key.pem
+    cert:./cert.pem
+    max-length:8192
+    threading:1
+}
+```
+
+It uses a key-value syntax, and uses a `:` as its seperator. A few key directives:  
+`host`, followed by a hostname signifies a host that will be available. Similar to nginx's `server_name` directive.  
+`globals` signifies all values that are of global importance, like the key and certificate file.  
+`directory` signifies the directory to look in for files on that specific host.  
+`index`, while not in the default config, signifies what path should be returned if the client only asks for `/` (or any subpaths without files).  
+`pesmode` signifies if the PES mode must be enabled, allowing the server to run custom Python code to manipulate the request or file further.  
+`block-ua` signifies if a specific (or loosely matched) User-Agent must be blocked from accessing the site.  
+`proxy` signifies if a the server needs to get the response from a different (remote) server but still needs to be available at this host.  
+`max-length` signifies the maximum length a request may have.  
+`threading` signifies if the threading engine should be enabled. This will lend more performance, but should be disabled if you use scripts that are not thread-safe.  
+AmethystConf has only 4 datatypes: `String`, `Boolean`, `Function` and `None`. A quick rundown:  
+`String` is the everything datatype. Everything is assumed to be a `String` unless it falls under the other categories.  
+`Boolean` is the datatype used to enable/disable features. A `Boolean` can have one of two possible values: `1` or `0`.  
+`Function` is the datatype used in `match()`, it signifies that the parser has to do some work on this string before it can use it.  
+`None` is the datatype assigned to any key without a value.  
+
+## PES (Python Extension Script)
+
+The PES (Python Extension Script) is one of Amethysts main selling points. It's a new type of a dynamic page. A PES file is pretty much a 
+Python script with some conventions. Currently it is in very alpha form. It will be heavily improved upon to make sure even people with no 
+Python knowledge can work with it. Here's how it works:  
+If PES mode is enabled, the request is sent to the `pes.py` script, more specifically, the `on_request(req)` function of the `PES()` class.  
+From there, the decoded request is given to you to play with. All of Amethysts request and file processing tools are available, and soon 
+a function will be added to hand the request back to Amethyst, if it is deemed not suitable for PES mode.  
+Once you're done manipulating the request, all you have to do is call `return self.build_response(http_status_code, resp_body, mimetype)` and Amethyst will handle the rest.  
+A few examples of what can be achieved with PES mode without writing any other language than Python, HTML and CSS:
+
+* Showing a random image from the `/pics` folder upon requesting `/randompic` from the server
+* Dynamically updating the time on a website
+* Create a complex calculator
+* Upload files to the server
+* Lock down webpages with a login prompt.
+* And much, much more.  
+
+While it might not be able to create truly dynamic pages (since PES runs server-side, not client-side), it is dynamically static, basically, it's a dynamic page until it's rendered
+in browser, where it's static, as PES cannot change anything there.  
+**WARNING!**
+PES is an advanced feature! You can absolutely compromise the security of your webserver by having a misconfigured PES file. While Amethyst still has a few protection measures built-in
+that activate before any request reaches the PES, but some are bypassed unless manually invoked in the PES. Because of that, here's a general user advisory:
+
+* Use `self.fh.read_file(file_path, host=None)` instead of `open(file_path)` because of file inclusion or directory traversal concerns.
+* Use `self.fh.write_file(file_path, host=None)` instead of `open(file_path)` because of file inclusion or directory traversal concerns.
+* **NEVER** allow the PES to run shell code!
+* **NEVER** allow the PES to run **ANYTHING** uploaded via the internet!
+* Try running as much of the code locally, getting data from the internet can not only take long, it can also pose a security risk.  
+
+The PES will **NOT** warn you if you have security issues, it's a very hands-off approach. The PES will happily run `sudo rm -rf / --no-preserve-root` if given the command and
+setup for shell execution and not tell you until everything is gone. Prevent those scenarios by limiting what PES does as much as possible!

amethyst.conf

@@ -0,0 +1,18 @@
+# WARNING: This is an alpha spec of NSCL 2.0!!
+
+host * {
+    directory:./html
+    pesmode:0
+    index:index.html
+}
+
+globals {
+    http:1
+    https:1
+    port:8080
+    https-port:8443
+    key:./key.pem
+    cert:./cert.pem
+    max-length:8192
+    threading:1
+}

amethyst.py

@@ -38,21 +38,23 @@ TODO: actually put normal comments in
 
 TODO: INPROG: add typing to all code, new code will feature it by default.
 """
-
+# Stable imports go here
+import sys
+import threading
 import os
 import mimetypes
 import threading
 import ssl
 import socket
-
-# import re
 import signal
-import sys
+# Experimental imports go here
+import select
+import subprocess
 
 try:
     if not os.getcwd() in sys.path:
         sys.path.append(f"{os.getcwd()}")
-    from certgen import AutoCertGen
+    from .certgen import AutoCertGen
 except ImportError:
     # just do nothing, it's not working anyway.
     print(
@@ -61,7 +63,7 @@ except ImportError:
     )
     # pass
 
-AMETHYST_BUILD_NUMBER = "0053"
+AMETHYST_BUILD_NUMBER = "0.3.1-0130-mt-tryout2"
 AMETHYST_REPO = "https://git.novacow.ch/Nova/PyWebServer/"
 
 
@@ -111,29 +113,31 @@ class ConfigParser:
 
     def query_config(self, key, host=None):
         if host:
-            return self.data["hosts"].get(host, {}).get(key)
-        if key == "hosts":
+            value = self.data["hosts"].get(host, {}).get(key)
+        elif key == "hosts":
             print(f"\n\n\nHosts!\nHosts: {self.data['hosts']}\n\n\n")
-            return list(self.data["hosts"].keys())
-        return self.data["globals"].get(key)
+            value = list(self.data["hosts"].keys())
+        else:
+            value = self.data["globals"].get(key)
+        if value == "0" or value == "1":
+            value = int(value)
+        return value
 
 
 class FileHandler:
-    CONFIG_FILE = "pywebsrv.conf"
-    new_conf = "new_conf.conf"
-
     def __init__(self, base_dir=None):
         # this is a fucking clusterfuck.
-        self.config_path = os.path.join(os.getcwd(), self.CONFIG_FILE)
-        self.new_conf = os.path.join(os.getcwd(), self.new_conf)
-        self.base_dir = self.read_config("directory")
-        with open(self.new_conf, "r") as f:
+        self.config_file = "amethyst.conf"
+        self.config_path = os.path.join(os.getcwd(), self.config_file)
+        with open(self.config_path, "r") as f:
             self.cfg = ConfigParser(f.read())
+        self.base_dir = self.read_config("directory")
         if not os.path.exists(self.config_path):
+            # uuh???
             print(
-                "The pywebsrv.conf file needs to be in the same directory "
-                "as pywebsrv.py! Get the default config file from:\n"
-                "https://git.novacow.ch/Nova/PyWebServer/raw/branch/main/pywebsrv.conf"
+                "The amethyst.conf file needs to be in the same directory "
+                "as amethyst.py! Get the default config file from:\n"
+                "https://git.novacow.ch/Nova/PyWebServer/raw/branch/2.0/amethyst.conf"
             )
             exit(1)
         # TODO: fix this please!!
@@ -141,7 +145,7 @@ class FileHandler:
     def read_file(self, file_path, directory=None):
         if "../" in file_path or "%" in file_path:
             return 403, None
-        if file_path == "api.py":
+        if file_path == "pes.py":
             return 404, None
 
         if directory is not None:
@@ -163,76 +167,11 @@ class FileHandler:
         if "../" in file_path or "%" in file_path:
             return 403
         full_path = os.path.join(self.base_dir, file_path.lstrip("/"))
-        with open(full_path, "a") as f:
+        with open(full_path, "wb") as f:
             f.write(data)
         return 0
 
-    def read_config(self, option):
-        """
-        clean code, whats that????
-        TODO: docs
-        """
-        option = option.lower()
-        valid_options = [
-            "port",
-            "directory",
-            "host",
-            "http",
-            "https",
-            "port-https",
-            "allow-localhost",
-            "disable-autocertgen",
-            "key-file",
-            "cert-file",
-            "block-ua",
-        ]
-        if option not in valid_options:
-            return None
-        with open(self.config_path, "r") as f:
-            for line in f:
-                if line.startswith("#"):
-                    continue
-                try:
-                    key, value = line.strip().split(":", 1)
-                except ValueError:
-                    return None
-                key = key.lower()
-                if key == option:
-                    if option == "host":
-                        seperated_values = value.split(",", -1)
-                        return [value.lower() for value in seperated_values]
-                    if option == "block-ua":
-                        seperated_values = value.split(",", -1)
-                        host_to_match = []
-                        literal_blocks = []
-                        for val in seperated_values:
-                            if val.startswith("match(") and val.endswith(")"):
-                                idx = val.index("(")
-                                idx2 = val.index(")")
-                                ua_to_match = val[idx + 1 : idx2]
-                                host_to_match.append(ua_to_match)
-                            else:
-                                literal_blocks.append(val)
-                        return host_to_match, literal_blocks
-                    if option == "port" or option == "port-https":
-                        return int(value)
-                    if (
-                        option == "http"
-                        or option == "https"
-                        or option == "allow-localhost"
-                        or option == "disable-autocertgen"
-                    ):
-                        return bool(int(value))
-                    if option == "directory":
-                        if value == "<Enter directory here>":
-                            return os.path.join(os.getcwd(), "html")
-                        if value.endswith("/"):
-                            value = value.rstrip("/")
-                        return value
-                    return value
-        return None
-
-    def read_new_config(self, key, host_name=None):
+    def read_config(self, key, host_name=None):
         print(
             f"\n\n\nQuery!\nkey: {key}\nhost_name: {host_name}\nret: {self.cfg.query_config(key, host_name)}"
         )
@@ -250,36 +189,58 @@ class FileHandler:
 class RequestParser:
     def __init__(self):
         self.file_handler = FileHandler()
-        self.hosts = self.file_handler.read_new_config("hosts")
+        self.hosts = self.file_handler.read_config("hosts")
         print(f"Hosts: {self.hosts}")
 
-    def parse_request_line(self, line, host):
+    def extract_header(self, header: str, request: bytes | str):
+        if isinstance(request, bytes):
+            request = request.decode("iso-8859-1", "ignore")
+        lines = request.splitlines()
+        for line in lines:
+            if line.startswith(header):
+                value = line.split(":")[1][1:]
+                return value
+        return None
+
+    def parse_request_line(self, line, host, no_mod=False):
         """Parses the HTTP request line."""
         try:
             method, path, version = line.split(" ")
         except ValueError:
-            return "DELETE", "/this/is/a/bogus/request", "HTTP/1.0"
-        if path.endswith("/") or ("." not in path):
+            return None, None, None
+        if (path.endswith("/") or ("." not in path)) and (not no_mod):
             if not path.endswith("/"):
                 path += "/"
-            index = self.file_handler.read_new_config("index", host) or "index.html"
+            index = self.file_handler.read_config("index", host) or "index.html"
             path += f"{index}"
         return method, path, version
 
+    def parse_match_blocks(self, to_parse: str | list):
+        if isinstance(to_parse, str):
+            to_parse = [to_parse]
+        match = []
+        literal = []
+        for block in to_parse:
+            if block.startswith('match("'):
+                adx = block[7:-2]
+                match.append(adx)
+            else:
+                literal.append(block)
+        return match, literal
+
     def ua_is_allowed(self, ua, host=None):
         """Parses and matches UA to block"""
+        # return True
+        _list = self.file_handler.read_config("block-ua", host)
+        if _list is None:
+            return True
+        match, literal = self.parse_match_blocks(_list)
+        if ua in literal:
+            return False
+        for _ua in match:
+            if _ua.lower() in ua.lower():
+                return False
         return True
-        # del host
-        # _list = self.file_handler.read_config("block-ua")
-        # if _list is None:
-        #     return True
-        # match, literal = self.file_handler.parse_match_blocks(_list)
-        # if ua in literal:
-        #     return False
-        # for _ua in match:
-        #     if _ua.lower() in ua.lower():
-        #         return False
-        # return True
 
     def is_method_allowed(self, method, host=None):
         """
@@ -288,11 +249,7 @@ class RequestParser:
         Falls back to allowing only 'GET' if the file does not exist.
         Should (for now) only be GET as I haven't implemented the logic for PUT
         """
-        # allowed_methods = ["GET"]
-        # While the logic for PUT, DELETE, etc. is not added, we shouldn't
-        # allow for it to attempt it.
-        # Prepatched for new update.
-        allowed_methods = self.file_handler.read_new_config("allowed-methods", host)
+        allowed_methods = self.file_handler.read_config("allowed-methods", host)
         if allowed_methods is None:
             allowed_methods = ["GET"]
         return method in allowed_methods
@@ -308,18 +265,154 @@ class RequestParser:
             host = host.rsplit(":", 1)[0]
         host = host.lstrip()
         host = host.rstrip()
-        if (
-            host == "localhost" or host == "127.0.0.1" or host == "[::1]"
-        ) and self.file_handler.read_new_config("allow-localhost"):
-            return True
         if self.hosts is None:
             return True
         if host not in self.hosts:
+            if "*" in self.hosts:
+                return "catchall"
             return False
         else:
             return True
 
 
+class ProxyServer:
+    def __init__(self, fh):
+        self.file_handler: FileHandler = fh
+        self.rq: RequestParser = RequestParser()
+
+    @staticmethod
+    def recv_all(sock):
+        chunks = []
+        while True:
+            try:
+                data = sock.recv(4096)
+                if not data:
+                    break
+                chunks.append(data)
+            except socket.timeout:
+                break
+        return b"".join(chunks)
+
+    def try_connection(
+        self, host: str, port: int, data: bytes, chost: str, force_tls: bool = None
+    ):
+        print(f"\n\n\nchost: {chost}\n\n\n")
+        nhost = self.file_handler.read_config("proxy", chost)
+        print(f"\n\n\nnhost: {nhost}\n\n\n")
+        # nhost will include http or https.
+        if nhost.startswith("https"):
+            nhost = nhost[6:-1]
+            do_tls = True
+        elif nhost.startswith("http"):
+            nhost = nhost[5:-1]
+            do_tls = False
+        else:
+            raise SyntaxError(
+                "Syntax error in config! Key: `proxy` Reason: `Expected http([...]) or https([...]), not "
+                f"{nhost[:6]}[...]{nhost[-1:]}!`"
+            )
+        if force_tls is True:
+            do_tls = True
+        print(f"\n\n\nnhost: {nhost}\n\n\n")
+        if ":" in nhost:
+            nport = int(nhost.split(":")[1])
+            nhost = nhost.split(":")[0]
+        else:
+            nport = port
+        print(f"{nhost}, {nport}, {data}")
+        data = self.reset_host(nhost, nport, data)
+        try:
+            print("Waiting on TCP start.")
+            return self.tcp_send(nhost, nport, data, do_tls)
+        except Exception as e:
+            raise Exception(f"Server replied unexpected. Reply from Python subsystem: {e}")
+
+    @staticmethod
+    def reset_host(host: str, port: int, data: bytes):
+        header_end = data.find(b"\r\n\r\n")
+
+        if header_end == -1:
+            return data
+
+        header_bytes = data[:header_end]
+        body = data[header_end + 4:]
+
+        headers = header_bytes.decode("iso-8859-1")
+
+        lines = headers.split("\r\n")
+        new_lines = []
+
+        for line in lines:
+            lower = line.lower()
+
+            if lower.startswith("host:"):
+                if port not in [80, 443]:
+                    line = f"Host: {host}:{port}"
+                else:
+                    line = f"Host: {host}"
+
+            elif lower.startswith("connection:"):
+                line = "Connection: close"
+
+            new_lines.append(line)
+
+        rebuilt_headers = "\r\n".join(new_lines).encode("iso-8859-1")
+
+        return rebuilt_headers + b"\r\n\r\n" + body
+
+    @staticmethod
+    def create_tls_context():
+        ctx = ssl.create_default_context()
+        ctx.check_hostname = False
+        ctx.verify_mode = ssl.CERT_NONE
+        return ctx
+
+    def tcp_send(self, host, port, data: bytes, do_tls: booll):
+        try:
+            with socket.create_connection((host, port), timeout=10) as raw_sock:
+                raw_sock.settimeout(10)
+                if do_tls:
+                    ctx = self.create_tls_context()
+                    server_hostname = host
+                    with ctx.wrap_socket(
+                        raw_sock, server_hostname=server_hostname
+                    ) as ssock:
+                        ssock.sendall(data)
+                        print("data reached")
+                        resp = self.recv_all(ssock)
+                        if self.rq.extract_header("Transfer-Encoding", resp) == "chunked":
+                            ssock.sendall(b"TRANSER-ENCODING IS NOT SUPPORTED")
+                            ssock.close()
+                            resp = (
+                                "HTTP/1.1 502 Gateway Error\r\nConnection: close\r\nContent-Length: 270\r\n"
+                                f"Server: Amethyst/build-{AMETHYST_BUILD_NUMBER}\r\n\r\n"
+                                "Gateway Error.\nThe upstream server tried to use a a transfer mode not "
+                                "yet supported\nExact error:\nE_DATA_STREAMING_NOT_SUPPORTED\n"
+                                "The unsupported mode:\nTransfer-Encoding: chunked.\n"
+                                "There is no fix. The problem lies with the proxy, and is not a fault of the upstream server."
+                            )
+                        return resp
+                else:
+                    print(f"\n\n\nraw data: {data}\n\n\n")
+                    raw_sock.sendall(data)
+                    print("Waiting for response...")
+                    resp = self.recv_all(raw_sock)
+                    if self.rq.extract_header("Transfer-Encoding", resp) is not None:
+                        raw_sock.sendall(b"TRANSER-ENCODING IS NOT SUPPORTED")
+                        raw_sock.close()
+                        resp = (
+                            "HTTP/1.1 502 Gateway Error\r\nConnection: close\r\nContent-Length: 270\r\n"
+                            f"Server: Amethyst/build-{AMETHYST_BUILD_NUMBER}\r\n\r\n"
+                            "Gateway Error.\nThe upstream server tried to use a a transfer mode not "
+                            "yet supported\nExact error:\nE_DATA_STREAMING_NOT_SUPPORTED\n"
+                            "The unsupported mode:\nTransfer-Encoding: chunked.\n"
+                            "There is no fix. The problem lies with the proxy, and is not a fault of the upstream server."
+                        )
+                    return resp
+        except Exception:
+            raise
+
+
 class WebServer:
     def __init__(
         self, http_port=8080, https_port=8443, cert_file="cert.pem", key_file="key.pem"
@@ -328,9 +421,12 @@ class WebServer:
         self.https_port = int(https_port)
         self.file_handler = FileHandler()
         self.parser = RequestParser()
-        self.cert_file = self.file_handler.read_new_config("cert") or cert_file
-        self.key_file = self.file_handler.read_new_config("key") or key_file
+        self.cert_file = self.file_handler.read_config("cert") or cert_file
+        self.key_file = self.file_handler.read_config("key") or key_file
+        self.max_length = int(self.file_handler.read_config("max-length")) or 8192
         self.skip_ssl = False
+        self.threading = bool(self.file_handler.read_config("threading"))
+        self.tlock = threading.Lock()
 
         # me when no certificate and key file
         if not os.path.exists(self.cert_file) or not os.path.exists(self.key_file):
@@ -358,9 +454,13 @@ class WebServer:
 
         self.http_socket = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
         self.http_socket.bind(("::", self.http_port))
+        # self.http_socket.settimeout(1)
 
-        self.https_socket = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
-        self.https_socket.bind(("::", self.https_port))
+        self.https_socket_raw = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
+        self.https_socket_raw.bind(("::", self.https_port))
+        # self.https_socket_raw.settimeout(1)
+
+        self.proxy_handler = ProxyServer(self.file_handler)
 
         if self.skip_ssl is False:
             # https gets the ssl treatment!! yaaaay :3
@@ -369,7 +469,7 @@ class WebServer:
                 certfile=self.cert_file, keyfile=self.key_file
             )
             self.https_socket = self.ssl_context.wrap_socket(
-                self.https_socket, server_side=True
+                self.https_socket_raw, server_side=True
             )
 
         self.http_404_html = (
@@ -393,58 +493,124 @@ class WebServer:
     def start(self, http, https):
         signal.signal(signal.SIGINT, self.shutdown)
         signal.signal(signal.SIGTERM, self.shutdown)
-
-        http_thread = threading.Thread(target=self.start_http, daemon=True)
-        https_thread = threading.Thread(target=self.start_https, daemon=True)
-
         if https is True:
             if self.skip_ssl is True:
                 print("WARN: You have enabled HTTPS without SSL!!")
                 yn = input("Is this intended behaviour? [y/N] ")
                 if yn.lower() == "n":
                     exit(1)
-            https_thread.start()
+            self.start_https()
+        else:
+            self.https_socket.close()
         if http is True:
-            http_thread.start()
-
-        http_thread.join()
-        https_thread.join()
+            self.start_http()
+        else:
+            self.http_socket.close()
 
     def start_http(self):
         self.http_socket.listen(5)
         print(f"HTTP server listening on port {self.http_port}...")
         while self.running:
             try:
+                ready, _, _ = select.select(
+                    [self.http_socket],
+                    [],
+                    [],
+                    1.0
+                )
+                if not ready:
+                    continue
                 conn, addr = self.http_socket.accept()
+                conn.settimeout(2)
+                if self.threading:
+                    threading.Thread(
+                        target=self.handle_connection,
+                        args=(conn, addr),
+                        daemon=True
+                    ).start()
+                else:
                     self.handle_connection(conn, addr)
+            except socket.timeout:
+                continue
+            except OSError as e:
+                if not self.running:
+                    break
+                print(f"OSError! {e}")
+                continue
             except Exception as e:
                 print(f"HTTP error: {e}")
-            except OSError:
-                break
 
     def start_https(self):
         self.https_socket.listen(5)
         print(f"HTTPS server listening on port {self.https_port}...")
         while self.running:
             try:
+                ready, _, _ = select.select(
+                    [self.https_socket],
+                    [],
+                    [],
+                    1.0
+                )
+                if not ready:
+                    continue
                 conn, addr = self.https_socket.accept()
+                conn.settimeout(2)
+                if self.threading:
+                    threading.Thread(
+                        target=self.handle_connection,
+                        args=(conn, addr),
+                        daemon=True
+                    ).start()
+                else:
                     self.handle_connection(conn, addr)
-            except Exception as e:
-                print(
-                    f"HTTPS error: {e}"
-                )  # be ready for ssl errors if you use a self-sign!!
-            except OSError:
+            except socket.timeout:
+                continue
+            except OSError as e:
+                if not self.running:
                     break
+                print(f"OSError! {e}")
+                continue
+            except Exception as e:
+                print(f"HTTPS error: {e}")
 
     def handle_connection(self, conn, addr):
         try:
-            data = conn.recv(512)
-            request = data.decode(errors="ignore")
+            data = b""
+            # Read headers
+            while b"\r\n\r\n" not in data:
+                chunk = conn.recv(4096)
+                if not chunk:
+                    break
+                data += chunk
+
+            headers, _, rest = data.partition(b"\r\n\r\n")
+
+            # Parse Content-Length
+            content_length = 0
+            for line in headers.split(b"\r\n"):
+                if line.lower().startswith(b"content-length:"):
+                    content_length = int(line.split(b":")[1].strip())
+
+            # print(f"Content-Length to server: {content_length}")
+
+            # Read body
+            body = rest
+            print(f"Rest length: {len(rest)}")
+            while len(body) < content_length:
+                chunk = conn.recv(4096)
+                # print(f"\n\nrecv returned {len(chunk)}\n\n")
+                if not chunk:
+                    print("\n\nsocket closed\n\n")
+                    break
+                body += chunk
+
+            data += body
+            request = data.decode("iso-8859-1", errors="ignore")
             if not data:
                 response = self.build_response(
                     400, "Bad Request"
                 )  # user did fucky-wucky
-            elif len(data) > 8192:
+            elif len(data) > self.max_length:
                 response = self.build_response(413, "Request too long")
             else:
                 response = self.handle_request(request, addr)
@@ -452,6 +618,7 @@ class WebServer:
             if isinstance(response, str):
                 response = response.encode()
 
+            print(len(response))
             conn.sendall(response)
         except Exception as e:
             print(f"Error handling connection: {e}")
@@ -466,7 +633,7 @@ class WebServer:
             conn.close()
 
     def handle_request(self, data, addr):
-        print(f"data: {data}")
+        # print(f"data: {data}")
         request_line = data.splitlines()[0]
 
         # Extract host from headers, never works though
@@ -474,6 +641,9 @@ class WebServer:
             if "Host" in line:
                 host = line.split(":", 1)[1].strip()
                 allowed = self.parser.host_parser(host)
+                if allowed == "catchall":
+                    host = "*"
+                    allowed = True
                 if not allowed:
                     return self.build_response(
                         403, "Connecting via this host is disallowed."
@@ -485,7 +655,7 @@ class WebServer:
         for line in data.splitlines():
             if "User-Agent" in line:
                 ua = line.split(":", 1)[1].strip()
-                allowed = self.parser.ua_is_allowed(ua)
+                allowed = self.parser.ua_is_allowed(ua, host)
                 if not allowed:
                     return self.build_response(
                         403, "This UA has been blocked by the owner of this site."
@@ -495,37 +665,72 @@ class WebServer:
             return self.build_response(400, "You cannot connect without a User-Agent.")
 
         if ":" in host:
-            host2 = host.rsplit(":", 1)[0]
+            host = host.rsplit(":", 1)[0]
         else:
-            host2 = host
+            host = host
 
-        method, path, version = self.parser.parse_request_line(request_line, host2)
+        method, path, version = self.parser.parse_request_line(request_line, host)
 
         if not all([method, path, version]):
             return self.build_response(400, "Bad Request")
 
+        if self.file_handler.read_config("proxy", host) is not None:
+            orig_host = host
+            value = self.file_handler.read_config("proxy", host)
+            if ":" in value:
+                host = value.split(":")[0]
+                port = int(value.split(":")[1][:-1])
+            else:
+                host = value
+                port = 443
+            return self.proxy_handler.try_connection(
+                host,
+                port,
+                data.encode("iso-8859-1"),
+                orig_host,
+            )
+
         # Figure out a better way to reload config
         if path == "/?pywebsrv_reload_conf=1":
             print("Got reload command! Reloading configuration...")
             self.file_handler = FileHandler()
             self.parser = RequestParser()
-            return self.build_response(302, "", host=host2)
+            return self.build_response(302, "", host=host)
 
         if not self.parser.is_method_allowed(method):
             return self.build_response(405, self.http_405_html)
 
         directory = (
-            self.file_handler.read_new_config("directory", host2)
+            self.file_handler.read_config("directory", host)
             or self.file_handler.base_dir
         )
 
-        if self.file_handler.read_new_config("apimode", host2) is True:
+        if bool(self.file_handler.read_config("pesmode", host)) is True:
             if not os.path.join(os.getcwd(), directory) in sys.path:
                 sys.path.append(f"{os.path.join(os.getcwd(), directory)}")
-            import api
+            import pes
+            try:
+                pesclass = pes.PES()
+                threadcompat = pesclass.THREAD_SAFETY
+                if not threadcompat and self.threading is True:
+                    print(
+                        "PES is not thread-safe yet threading is enabled!\n"
+                        "Amethyst CANNOT guarantee data intergity!\n"
+                        "It is HIGHLY recommended you make your script thread-safe!\n"
+                    )
+                    with self.tlock:
+                        return pesclass.on_request(data)
+                return pesclass.on_request(data)
+            except Exception as e:
+                return self.build_response(
+                    500,
+                    "Amethyst is currently unable to serve your request. Below is debug info.\r\n"
+                    f"Error: {e}; Version: amethyst-b{AMETHYST_BUILD_NUMBER}\r\n"
+                    "You cannot do anything at this time, the server owner has made a misconfiguration in their Python Extension Script",
+                )
 
-            apiclass = api.API()
-            return apiclass.on_request(data)
+        if bool(self.file_handler.read_config("621mode", host)) is True:
+            return self.build_response(621, "")
 
         file_content, mimetype = self.file_handler.read_file(path, directory)
 
@@ -625,14 +830,14 @@ class WebServer:
 
         if status_code == 621:
             headers = (
-                f"HTTP/1.1 {status_code} {status_message}\r\n"
+                "HTTP/1.1 302 UwU Nya!\r\n"
                 "Server: Amethyst/build-0621\r\n"
-                "Content-Length: 30\r\n"
-                f"Connection: close\r\n\r\n"
-            )
-            body = "https://e621.net/posts/6155664"
-
-        print(f"{headers + body}")
+                "Content-Length: 0\r\n"
+                "Connection: close\r\n"
+                "Note: congrats, you found a funny. i guess.\r\n"
+                "Host: https://e621.net/posts/\r\n\r\n"
+            ).encode("iso-8859-1")
+            body = "".encode("iso-8859-1")
         return headers + body
 
     def shutdown(self, signum, frame):
@@ -640,7 +845,6 @@ class WebServer:
         self.running = False
         self.http_socket.close()
         self.https_socket.close()
-        sys.exit(0)
 
 
 def main():
@@ -657,11 +861,11 @@ def main():
     input("Press <Enter> to continue. ")
     file_handler = FileHandler()
     file_handler.base_dir = file_handler.read_config("directory")
-    http_port = file_handler.read_new_config("port") or 8080
-    https_port = file_handler.read_new_config("https-port") or 8443
-    http_enabled = bool(file_handler.read_new_config("http")) or True
+    http_port = file_handler.read_config("port")
+    https_port = file_handler.read_config("https-port")
+    http_enabled = bool(file_handler.read_config("http"))
     print(http_enabled)
-    https_enabled = bool(file_handler.read_new_config("https")) or False
+    https_enabled = bool(file_handler.read_config("https"))
     print(https_enabled)
     server = WebServer(http_port=http_port, https_port=https_port)
     server.start(http_enabled, https_enabled)

certgen.py

@@ -6,28 +6,50 @@ import datetime
 
 
 class AutoCertGen:
-    def __init__(self):
-        pass
+    def __init__(self, name="website", org="organization", locale="place", province="province", country="ZZ", dns="localhost"):
+        self.name = name
+        self.org = org
+        self.locale = locale
+        self.province = province
+        self.country = country
+        self.dns = dns
 
-    def gen_cert(self):
-        # Generate private key
+    def generate_self_signed_cert(self, different_issuer=False):
         private_key = rsa.generate_private_key(
             public_exponent=65537,
             key_size=2048,
         )
 
-        # Define subject and issuer (self-signed)
+        if different_issuer is True:
+            subject = x509.Name(
+                [
+                    x509.NameAttribute(NameOID.COUNTRY_NAME, self.country),
+                    x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, self.province),
+                    x509.NameAttribute(NameOID.LOCALITY_NAME, self.locale),
+                    x509.NameAttribute(NameOID.ORGANIZATION_NAME, self.org),
+                    x509.NameAttribute(NameOID.COMMON_NAME, self.name),
+                ]
+            )
+            issuer = x509.Name(
+                [
+                    x509.NameAttribute(NameOID.COUNTRY_NAME, "RU"),
+                    x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "Красноярск Область"),
+                    x509.NameAttribute(NameOID.LOCALITY_NAME, "Красноярск"),
+                    x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Amethyst Group"),
+                    x509.NameAttribute(NameOID.COMMON_NAME, "Amethyst Untrusted Signing Certificate"),
+                ]
+            )
+        else:
             subject = issuer = x509.Name(
                 [
-                x509.NameAttribute(NameOID.COUNTRY_NAME, "ZZ"),
-                x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "Some province"),
-                x509.NameAttribute(NameOID.LOCALITY_NAME, "Some place"),
-                x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Some org"),
-                x509.NameAttribute(NameOID.COMMON_NAME, "localhost"),
+                    x509.NameAttribute(NameOID.COUNTRY_NAME, self.country),
+                    x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, self.province),
+                    x509.NameAttribute(NameOID.LOCALITY_NAME, self.locale),
+                    x509.NameAttribute(NameOID.ORGANIZATION_NAME, self.org),
+                    x509.NameAttribute(NameOID.COMMON_NAME, self.name),
                 ]
             )
 
-        # Create certificate
         certificate = (
             x509.CertificateBuilder()
             .subject_name(subject)
@@ -39,12 +61,11 @@ class AutoCertGen:
                 datetime.datetime.utcnow() + datetime.timedelta(days=365)
             )  # 1 year validity
             .add_extension(
-                x509.SubjectAlternativeName([x509.DNSName("localhost")]), critical=False
+                x509.SubjectAlternativeName([x509.DNSName(self.dns)]), critical=False
             )
             .sign(private_key, hashes.SHA256())
         )
 
-        # Save private key
         with open("key.pem", "wb") as f:
             f.write(
                 private_key.private_bytes(
@@ -54,8 +75,5 @@ class AutoCertGen:
                 )
             )
 
-        # Save certificate
         with open("cert.pem", "wb") as f:
             f.write(certificate.public_bytes(serialization.Encoding.PEM))
-
-        print("Self-signed certificate and private key generated for HTTPS server!")

html/index.html

@@ -8,7 +8,7 @@
     <h1>Hello from Amethyst!</h1>
     <h2>This page confirms Amethyst can read files from your PC or server and serve them to your browser!</h2>
     <p>This is a test page, if you aren't the server owner, they might not have finished setting up their site, be patient. If this doesn't go away after a while, tell them they've made an oopsie</p>
-    <p>This server runs Amethyst Pre-Rel Build 0053</p>
+    <p>This server runs Amethyst build 0.3.0-0114-mt-tryout1</p>
     </center>
 </body>
 </html>

html/pes.py

@@ -9,10 +9,10 @@ import os
 
 if not os.getcwd() in sys.path:
     sys.path.append(os.getcwd())
-import pywebsrv
+import amethyst
 
 
-class API:
+class PES:
     """
     class
     """
@@ -20,7 +20,10 @@ class API:
     def __init__(self):
         # DO NOT USE THIS CLASS FOR PROGRAM, ONLY ON_REQUEST PLEASE!!
         # Below go definitions to get things working.
-        self.build_response = pywebsrv.WebServer.build_binary_response
+        self.build_response = amethyst.WebServer.build_binary_response
+        self.fh = amethyst.FileHandler("..")
+        self.rq = amethyst.RequestParser()
+        self.THREAD_SAFETY: bool = True
 
     def on_request(self, req):
-        return self.build_response(200, "This is a test", "text/html")
+        return self.build_response(200, b"Heyhey! This is the default PES script!", "text/html")

new_conf.conf

@@ -1,32 +0,0 @@
-# WARNING: This is an alpha spec of NSCL 2.0!!
-
-host 192.168.2.196 {
-    directory:/home/nova/Downloads/test/html
-    allowed-methods:GET
-    block-ua:match("Discordbot"),match("Google")
-}
-
-host localhost {
-    directory:/home/nova/PyWebServer/html2
-    allowed-methods:GET,PUT
-    block-ip:10.1.100.2
-    apimode:0
-    block-ua:match("Discordbot")
-}
-
-host 192.168.1.213 {
-    directory:/home/nova/PyWebServer/html
-    allowed-methods:GET,PUT
-    block-ip:10.1.100.2
-    block-ua:match("Discordbot")
-}
-
-globals {
-    http:1
-    https:1
-    port:8080
-    https-port:8443
-    allow-localhost:1
-    key:/home/nova/PyWebServer/ssl/key.pem
-    cert:/home/nova/PyWebServer/ssl/cert.pem
-}

pywebsrv.conf

@@ -1,33 +0,0 @@
-# Using NSCL 1.3
-# Port defenition. What ports to use.
-# port is the HTTP port, port-https is the HTTPS port
-port:8080
-port-https:8443
-# Here you choose what directory PyWebServer looks in for files.
-directory:/home/nova/PyWebServer/html
-# Host defenition, what hosts you can connect via.
-# You can use FQDNs, IP-addresses and localhost,
-# Support for multiple hosts is coming.
-host:localhost,10.185.213.118
-# Enables HTTP support. (Only enables/disables the HTTP port.)
-http:1
-# Enables HTTPS support. (Only enables/disables the HTTPS port.)
-https:1
-# Allows the use of localhost to connect.
-# The default is on, this is seperate of the host defenition.
-allow-localhost:1
-# If you're using the webserver in a library form,
-# you can disable the AutoCertGen and never trigger it.
-disable-autocertgen:0
-# If you wish to block IP-addresses, this function is coming though.
-# block-ip:0.0.0.0,1.1.1.1,2.2.2.2
-# If you wish to block User-Agents.
-block-ua:match(Discordbot),match(google)
-
-# TEST: experimental non-defined keys go here:
-# keyfile key
-key-file:/home/nova/PyWebServer/key.pem
-# certfile keys
-cert-file:/home/nova/PyWebServer/cert.pem
-# allowed-methods, csv's
-allowed-methods:GET