Merge branch '2.0' of https://git.novacow.ch/Nova/PyWebServer into 2.0

Actually working properly!

CHANGELOG.md

@@ -12,7 +12,7 @@
 
 * Fixed bugs pertaining to proxy
 
-* Attempted fix at hanging socket by introducing a default 25 second timeout.
+* Attempted fix at hanging socket by introducing a default 2 second timeout.
 
 ## Configuration changes
 

PES.md

@@ -1,7 +1,11 @@
 # PES
+
 This is a quick reference document on how to implement PES.
+
 ## Example script:
+
 The default script is the following:
+
 ```python
 import sys
 import os
@@ -12,26 +16,28 @@ import amethyst
 
 
 class PES:
-    """
-    class
-    """
-
     def __init__(self):
-        # DO NOT USE THIS CLASS FOR PROGRAM, ONLY ON_REQUEST PLEASE!!
-        # Below go definitions to get things working.
+        # DO NOT USE THIS FUNCTION FOR PROGRAM, ONLY ON_REQUEST PLEASE!!
+        # WARNING: ONLY CHANGE THE THREAD_SAFETY VARIABLE! DO NOT ADD OR REMOVE ANYTHING!
+        # THEY WILL COMPROMISE ANY THREAD-SAFETY SECURITY MECHANISMS AMETHYST HAS IN PLACE!
+        # YOU HAVE BEEN WARNED!
         self.build_response = amethyst.WebServer.build_binary_response
         self.fh = amethyst.FileHandler("..")
         self.rq = amethyst.RequestParser()
-        # NOTE: THREAD_SAFETY is a required setting, as it defines
-        # if it can run within the Amethyst thread pool or needs to
-        # run independently
-        # False = run independent. True = run in Amethyst thread pool.
-        self.THREAD_SAFETY: bool = False
+        self.THREAD_SAFETY: bool = True
 
     def on_request(self, req):
-        return self.build_response(200, "This is a test", "text/html")
-
-if __name__ == "__main__":
-    # Code to run if it is not thread-safe.
-    PES.on_request(PES, "request")
+        return self.build_response(200, b"Heyhey! This is the default PES script!", "text/html")
 ```
+
+## Threading and Thread-safety.
+
+When you execute PES scripts, they have a high chance of not being thread-safe because they edit the page you visit. It means you cannot guarantee the data you want is actually the data getting sent over the wire. This issue can be fixed in multiple ways:
+
+1. Disabling threading, then only one script can run at once, but this costs a lot of performance
+
+2. Enforcing thread-safety on all scripts, this will mean every script can be trusted, but makes development difficult (especially for people with no coding experience)
+
+3. Implementing mechanisms that will try to patch up holes if threading is enabled and a thread-unsafe script is loaded.
+
+Amethyst uses fix 3. It only allows one script to run at a time (if both threading is enabled and a thread-unsafe script is loaded), but if you specify only one host to use PES, all other hosts still enjoy the benefits of a multithreaded server! This makes sure that if user 1 and user 2 both request something and PES is involved, user 1 receives part of the data they want and part of the data user 2 wants and vice versa. This security mechanism only works if you respect them. Amethyst will throw a warning if you have a situation you have an unsafe script and run threaded, this warning isn't critical, as the script will still execute with security mechanisms, but the warning in the script is clear. Amethyst can't help if you make it explicitly unsafe yourself.

README.md

@@ -9,15 +9,17 @@ Once a milestone is hit (e.g. a new feature fully implemented), I'll publish a r
 
 ## Approaching 1.0.0!
 
-Amethyst is finally approaching 1.0.0! Very very soon I will feature-freeze the project and begin just fixing bugs and cleaning up code! This may take a bit because the codebase is very cluttered, and because all features are there in a basic state, it would be better to fix and clean up what I have, so I have a workable codebase for implementing new features, and because new features aren't going to be added anyway, I might as well fully release the project!
+I have feature-freezed the project! Any further release is just cleaning up the codebase and getting some snazzy new default webpage or something idk yet xdd.
 
 ## Currently working features:
 
-* New configuration is ~95% done, most features work.
+* New configuration!
 * Fixed **A LOT** of unreported bugs from the old code.
 * More resilliency against errors.
 * Improved security.
 * Proxy almost working!
+* Multithreading!
+* Python Execution Script!
 
 ## Project status:
 

amethyst.py

@@ -38,16 +38,17 @@ TODO: actually put normal comments in
 
 TODO: INPROG: add typing to all code, new code will feature it by default.
 """
-
+# Stable imports go here
+import sys
+import threading
 import os
 import mimetypes
 import threading
 import ssl
 import socket
-
-# import re
 import signal
-import sys
+import select
+# Experimental imports go here
 
 try:
     if not os.getcwd() in sys.path:
@@ -61,7 +62,7 @@ except ImportError:
     )
     # pass
 
-AMETHYST_BUILD_NUMBER = "0.3.0-0114-mt-tryout1"
+AMETHYST_BUILD_NUMBER = "0.99.0-0134-ff"
 AMETHYST_REPO = "https://git.novacow.ch/Nova/PyWebServer/"
 
 
@@ -113,7 +114,6 @@ class ConfigParser:
         if host:
             value = self.data["hosts"].get(host, {}).get(key)
         elif key == "hosts":
-            print(f"\n\n\nHosts!\nHosts: {self.data['hosts']}\n\n\n")
             value = list(self.data["hosts"].keys())
         else:
             value = self.data["globals"].get(key)
@@ -170,9 +170,6 @@ class FileHandler:
         return 0
 
     def read_config(self, key, host_name=None):
-        print(
-            f"\n\n\nQuery!\nkey: {key}\nhost_name: {host_name}\nret: {self.cfg.query_config(key, host_name)}"
-        )
         return self.cfg.query_config(key, host_name)
 
     def autocert(self):
@@ -188,7 +185,6 @@ class RequestParser:
     def __init__(self):
         self.file_handler = FileHandler()
         self.hosts = self.file_handler.read_config("hosts")
-        print(f"Hosts: {self.hosts}")
 
     def extract_header(self, header: str, request: bytes | str):
         if isinstance(request, bytes):
@@ -258,7 +254,6 @@ class RequestParser:
         Mfw im in an ugly code writing contest and my opponent is nova while writing a side project
         """
         host = f"{host}"
-        print(f"hosts: {self.hosts}, host: {host}, split: {host.rsplit(':', 1)[0]}")
         if ":" in host:
             host = host.rsplit(":", 1)[0]
         host = host.lstrip()
@@ -294,9 +289,7 @@ class ProxyServer:
     def try_connection(
         self, host: str, port: int, data: bytes, chost: str, force_tls: bool = None
     ):
-        print(f"\n\n\nchost: {chost}\n\n\n")
         nhost = self.file_handler.read_config("proxy", chost)
-        print(f"\n\n\nnhost: {nhost}\n\n\n")
         # nhost will include http or https.
         if nhost.startswith("https"):
             nhost = nhost[6:-1]
@@ -311,16 +304,13 @@ class ProxyServer:
             )
         if force_tls is True:
             do_tls = True
-        print(f"\n\n\nnhost: {nhost}\n\n\n")
         if ":" in nhost:
             nport = int(nhost.split(":")[1])
             nhost = nhost.split(":")[0]
         else:
             nport = port
-        print(f"{nhost}, {nport}, {data}")
         data = self.reset_host(nhost, nport, data)
         try:
-            print("Waiting on TCP start.")
             return self.tcp_send(nhost, nport, data, do_tls)
         except Exception as e:
             raise Exception(f"Server replied unexpected. Reply from Python subsystem: {e}")
@@ -376,7 +366,6 @@ class ProxyServer:
                         raw_sock, server_hostname=server_hostname
                     ) as ssock:
                         ssock.sendall(data)
-                        print("data reached")
                         resp = self.recv_all(ssock)
                         if self.rq.extract_header("Transfer-Encoding", resp) == "chunked":
                             ssock.sendall(b"TRANSER-ENCODING IS NOT SUPPORTED")
@@ -391,9 +380,7 @@ class ProxyServer:
                             )
                         return resp
                 else:
-                    print(f"\n\n\nraw data: {data}\n\n\n")
                     raw_sock.sendall(data)
-                    print("Waiting for response...")
                     resp = self.recv_all(raw_sock)
                     if self.rq.extract_header("Transfer-Encoding", resp) is not None:
                         raw_sock.sendall(b"TRANSER-ENCODING IS NOT SUPPORTED")
@@ -423,6 +410,8 @@ class WebServer:
         self.key_file = self.file_handler.read_config("key") or key_file
         self.max_length = int(self.file_handler.read_config("max-length")) or 8192
         self.skip_ssl = False
+        self.threading = bool(self.file_handler.read_config("threading"))
+        self.tlock = threading.Lock()
 
         # me when no certificate and key file
         if not os.path.exists(self.cert_file) or not os.path.exists(self.key_file):
@@ -450,11 +439,9 @@ class WebServer:
 
         self.http_socket = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
         self.http_socket.bind(("::", self.http_port))
-        self.http_socket.settimeout(25)
 
         self.https_socket_raw = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
         self.https_socket_raw.bind(("::", self.https_port))
-        self.https_socket_raw.settimeout(25)
 
         self.proxy_handler = ProxyServer(self.file_handler)
 
@@ -487,27 +474,45 @@ class WebServer:
         self.running = True
 
     def start(self, http, https):
+        signal.signal(signal.SIGINT, self.shutdown)
+        signal.signal(signal.SIGTERM, self.shutdown)
+
+        http_thread = threading.Thread(target=self.start_http, daemon=True)
+        https_thread = threading.Thread(target=self.start_https, daemon=True)
+
         if https is True:
             if self.skip_ssl is True:
                 print("WARN: You have enabled HTTPS without SSL!!")
                 yn = input("Is this intended behaviour? [y/N] ")
                 if yn.lower() == "n":
                     exit(1)
-            self.start_https()
+            https_thread.start()
         else:
             self.https_socket.close()
         if http is True:
-            self.start_http()
+            http_thread.start()
         else:
             self.http_socket.close()
 
+        http_thread.join()
+        https_thread.join()
+
     def start_http(self):
         self.http_socket.listen(5)
         print(f"HTTP server listening on port {self.http_port}...")
         while self.running:
             try:
+                ready, _, _ = select.select(
+                    [self.http_socket],
+                    [],
+                    [],
+                    1.0
+                )
+                if not ready:
+                    continue
                 conn, addr = self.http_socket.accept()
-                if self.file_handler.read_config("threading") is True:
+                conn.settimeout(2)
+                if self.threading:
                     threading.Thread(
                         target=self.handle_connection,
                         args=(conn, addr),
@@ -515,19 +520,31 @@ class WebServer:
                     ).start()
                 else:
                     self.handle_connection(conn, addr)
-            except OSError:
-                break
+            except socket.timeout:
+                continue
+            except OSError as e:
+                if not self.running:
+                    break
+                continue
             except Exception as e:
-                if not "timeout" in f"{e}":
-                    print(f"HTTP error: {e}")
+                print(f"HTTP error: {e}")
 
     def start_https(self):
         self.https_socket.listen(5)
         print(f"HTTPS server listening on port {self.https_port}...")
         while self.running:
             try:
+                ready, _, _ = select.select(
+                    [self.https_socket],
+                    [],
+                    [],
+                    1.0
+                )
+                if not ready:
+                    continue
                 conn, addr = self.https_socket.accept()
-                if self.file_handler.read_config("threading") is True:
+                conn.settimeout(2)
+                if self.threading:
                     threading.Thread(
                         target=self.handle_connection,
                         args=(conn, addr),
@@ -535,11 +552,14 @@ class WebServer:
                     ).start()
                 else:
                     self.handle_connection(conn, addr)
-            except OSError:
-                break
+            except socket.timeout:
+                continue
+            except OSError as e:
+                if not self.running:
+                    break
+                continue
             except Exception as e:
-                if not "timeout" in f"{e}":
-                    print(f"HTTPS error: {e}")
+                print(f"HTTPS error: {e}")
 
     def handle_connection(self, conn, addr):
         try:
@@ -559,14 +579,10 @@ class WebServer:
                 if line.lower().startswith(b"content-length:"):
                     content_length = int(line.split(b":")[1].strip())
 
-            # print(f"Content-Length to server: {content_length}")
-
             # Read body
             body = rest
-            print(f"Rest length: {len(rest)}")
             while len(body) < content_length:
                 chunk = conn.recv(4096)
-                # print(f"\n\nrecv returned {len(chunk)}\n\n")
                 if not chunk:
                     print("\n\nsocket closed\n\n")
                     break
@@ -586,7 +602,6 @@ class WebServer:
             if isinstance(response, str):
                 response = response.encode()
 
-            print(len(response))
             conn.sendall(response)
         except Exception as e:
             print(f"Error handling connection: {e}")
@@ -601,10 +616,8 @@ class WebServer:
             conn.close()
 
     def handle_request(self, data, addr):
-        # print(f"data: {data}")
         request_line = data.splitlines()[0]
 
-        # Extract host from headers, never works though
         for line in data.splitlines():
             if "Host" in line:
                 host = line.split(":", 1)[1].strip()
@@ -680,7 +693,14 @@ class WebServer:
             try:
                 pesclass = pes.PES()
                 threadcompat = pesclass.THREAD_SAFETY
-                # if not threadcompat:
+                if not threadcompat and self.threading is True:
+                    print(
+                        "PES is not thread-safe yet threading is enabled!\n"
+                        "Amethyst CANNOT guarantee data intergity!\n"
+                        "It is HIGHLY recommended you make your script thread-safe!\n"
+                    )
+                    with self.tlock:
+                        return pesclass.on_request(data)
                 return pesclass.on_request(data)
             except Exception as e:
                 return self.build_response(
@@ -696,7 +716,6 @@ class WebServer:
         file_content, mimetype = self.file_handler.read_file(path, directory)
 
         if file_content == 403:
-            print("WARN: Directory traversal attack prevented.")  # look ma, security!!
             return self.build_response(403, self.http_403_html)
         if file_content == 404:
             return self.build_response(404, self.http_404_html)
@@ -750,7 +769,6 @@ class WebServer:
             200: "OK",
             204: "No Content",
             302: "Found",
-            304: "Not Modified",  # TODO KEKL
             400: "Bad Request",
             403: "Forbidden",
             404: "Not Found",
@@ -764,8 +782,6 @@ class WebServer:
         if isinstance(body, str):
             body = body.encode()
 
-        # TODO: dont encode yet, and i encode. awesome comments here.
-        # Don't encode yet, if 302 status code we have to include location.
         headers = (
             f"HTTP/1.1 {status_code} {status_message}\r\n"
             f"Server: Amethyst/build-{AMETHYST_BUILD_NUMBER}\r\n"
@@ -774,13 +790,6 @@ class WebServer:
         ).encode()
 
         if status_code == 302:
-            # 302 currently only happens when the reload is triggered.
-            # Why not 307, Moved Permanently? Because browsers will cache the
-            # response and not send the reload command.
-            # if port == 443:
-            #     host = f"https://{host}/"
-            # else:
-            #     host = f"http://{host}/"
             headers = (
                 f"HTTP/1.1 {status_code} {status_message}\r\n"
                 f"Location: {host}\r\n"
@@ -806,7 +815,6 @@ class WebServer:
         self.running = False
         self.http_socket.close()
         self.https_socket.close()
-        sys.exit(0)
 
 
 def main():
@@ -826,9 +834,7 @@ def main():
     http_port = file_handler.read_config("port")
     https_port = file_handler.read_config("https-port")
     http_enabled = bool(file_handler.read_config("http"))
-    print(http_enabled)
     https_enabled = bool(file_handler.read_config("https"))
-    print(https_enabled)
     server = WebServer(http_port=http_port, https_port=https_port)
     server.start(http_enabled, https_enabled)