diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..957351c
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,21 @@
+# Changelog from 0.2.0 to 0.3.0
+
+## Major changes
+
+* Improved multithreading engine to be actually multithreaded
+
+* Gave the Python extension a beter name
+
+* Getting ready for feature-freeze.
+
+## Minor changes
+
+* Fixed bugs pertaining to proxy
+
+* Attempted fix at hanging socket by introducing a default 25 second timeout.
+
+## Configuration changes
+
+* Added a new `threading` key to define if threading should be enabled.
+
+
diff --git a/PES.md b/PES.md
new file mode 100644
index 0000000..bdc42fe
--- /dev/null
+++ b/PES.md
@@ -0,0 +1,37 @@
+# PES
+This is a quick reference document on how to implement PES.
+## Example script:
+The default script is the following:
+```python
+import sys
+import os
+
+if not os.getcwd() in sys.path:
+ sys.path.append(os.getcwd())
+import amethyst
+
+
+class PES:
+ """
+ class
+ """
+
+ def __init__(self):
+ # DO NOT USE THIS CLASS FOR PROGRAM, ONLY ON_REQUEST PLEASE!!
+ # Below go definitions to get things working.
+ self.build_response = amethyst.WebServer.build_binary_response
+ self.fh = amethyst.FileHandler("..")
+ self.rq = amethyst.RequestParser()
+ # NOTE: THREAD_SAFETY is a required setting, as it defines
+ # if it can run within the Amethyst thread pool or needs to
+ # run independently
+ # False = run independent. True = run in Amethyst thread pool.
+ self.THREAD_SAFETY: bool = False
+
+ def on_request(self, req):
+ return self.build_response(200, "This is a test", "text/html")
+
+if __name__ == "__main__":
+ # Code to run if it is not thread-safe.
+ PES.on_request(PES, "request")
+```
diff --git a/README.md b/README.md
index 9b549a5..ddc9afd 100644
--- a/README.md
+++ b/README.md
@@ -1,12 +1,18 @@
# Amethyst Web Server
## A word of warning!
+
Currently Amethyst is in very early alpha stage, a lot of things will be broken, names won't be correct,
promised features missing, but I'm very much working on it live!
Every save I do increments the build number by 1, I won't publish all of them, but some of them will be published.
Once a milestone is hit (e.g. a new feature fully implemented), I'll publish a release!
+## Approaching 1.0.0!
+
+Amethyst is finally approaching 1.0.0! Very very soon I will feature-freeze the project and begin just fixing bugs and cleaning up code! This may take a bit because the codebase is very cluttered, and because all features are there in a basic state, it would be better to fix and clean up what I have, so I have a workable codebase for implementing new features, and because new features aren't going to be added anyway, I might as well fully release the project!
+
## Currently working features:
+
* New configuration is ~95% done, most features work.
* Fixed **A LOT** of unreported bugs from the old code.
* More resilliency against errors.
@@ -14,16 +20,20 @@ Once a milestone is hit (e.g. a new feature fully implemented), I'll publish a r
* Proxy almost working!
## Project status:
+
Amethyst will stay in beta for a while, I want all features to work, but I will make pre-release versions that are mostly stable.
They can be found as the `amethyst-prerel-0.a.b` releases. I won't guarantee 100% stability, but waay more than just some random build.
## Install instructions:
+
Install Python, execute `amethyst.py` and change the provided config.
## Minimum requirements:
+
Python 3.10+
And whatever PC that happens to run that.
I recommend Python 3.12 or above though, with a PC running:
+
* Windows 8.1+
* macOS 10.15+
* Linux 4.19+
@@ -31,14 +41,15 @@ I recommend Python 3.12 or above though, with a PC running:
* Some other somewhat recent OS.
## The webserver itself:
+
The Amethyst webserver is meant to be easy to use and configure. Its configuration takes inspiration from nginx and Caddyfile.
The language the configuration is made in is AmethystConf.
The default config is as follows:
+
```amethystconf
host * {
directory:./html
pesmode:0
- block-ua:match("Discordbot")
index:index.html
}
@@ -50,8 +61,10 @@ globals {
key:./key.pem
cert:./cert.pem
max-length:8192
+ threading:1
}
```
+
It uses a key-value syntax, and uses a `:` as its seperator. A few key directives:
`host`, followed by a hostname signifies a host that will be available. Similar to nginx's `server_name` directive.
`globals` signifies all values that are of global importance, like the key and certificate file.
@@ -60,13 +73,16 @@ It uses a key-value syntax, and uses a `:` as its seperator. A few key directive
`pesmode` signifies if the PES mode must be enabled, allowing the server to run custom Python code to manipulate the request or file further.
`block-ua` signifies if a specific (or loosely matched) User-Agent must be blocked from accessing the site.
`proxy` signifies if a the server needs to get the response from a different (remote) server but still needs to be available at this host.
-`max-length` signifies the maximum length a request may have.
+`max-length` signifies the maximum length a request may have.
+`threading` signifies if the threading engine should be enabled. This will lend more performance, but should be disabled if you use scripts that are not thread-safe.
AmethystConf has only 4 datatypes: `String`, `Boolean`, `Function` and `None`. A quick rundown:
`String` is the everything datatype. Everything is assumed to be a `String` unless it falls under the other categories.
`Boolean` is the datatype used to enable/disable features. A `Boolean` can have one of two possible values: `1` or `0`.
`Function` is the datatype used in `match()`, it signifies that the parser has to do some work on this string before it can use it.
`None` is the datatype assigned to any key without a value.
+
## PES (Python Extension Script)
+
The PES (Python Extension Script) is one of Amethysts main selling points. It's a new type of a dynamic page. A PES file is pretty much a
Python script with some conventions. Currently it is in very alpha form. It will be heavily improved upon to make sure even people with no
Python knowledge can work with it. Here's how it works:
@@ -75,6 +91,7 @@ From there, the decoded request is given to you to play with. All of Amethysts r
a function will be added to hand the request back to Amethyst, if it is deemed not suitable for PES mode.
Once you're done manipulating the request, all you have to do is call `return self.build_response(http_status_code, resp_body, mimetype)` and Amethyst will handle the rest.
A few examples of what can be achieved with PES mode without writing any other language than Python, HTML and CSS:
+
* Showing a random image from the `/pics` folder upon requesting `/randompic` from the server
* Dynamically updating the time on a website
* Create a complex calculator
@@ -87,6 +104,7 @@ in browser, where it's static, as PES cannot change anything there.
**WARNING!**
PES is an advanced feature! You can absolutely compromise the security of your webserver by having a misconfigured PES file. While Amethyst still has a few protection measures built-in
that activate before any request reaches the PES, but some are bypassed unless manually invoked in the PES. Because of that, here's a general user advisory:
+
* Use `self.fh.read_file(file_path, host=None)` instead of `open(file_path)` because of file inclusion or directory traversal concerns.
* Use `self.fh.write_file(file_path, host=None)` instead of `open(file_path)` because of file inclusion or directory traversal concerns.
* **NEVER** allow the PES to run shell code!
diff --git a/amethyst.conf b/amethyst.conf
index a897979..0d3c950 100644
--- a/amethyst.conf
+++ b/amethyst.conf
@@ -2,9 +2,8 @@
host * {
directory:./html
- apimode:0
- block-ua:match("Discordbot")
- index:index2.html
+ pesmode:0
+ index:index.html
}
globals {
@@ -15,4 +14,5 @@ globals {
key:./key.pem
cert:./cert.pem
max-length:8192
+ threading:1
}
diff --git a/amethyst.py b/amethyst.py
index 4ed2f52..ec08924 100644
--- a/amethyst.py
+++ b/amethyst.py
@@ -61,7 +61,7 @@ except ImportError:
)
# pass
-AMETHYST_BUILD_NUMBER = "0083"
+AMETHYST_BUILD_NUMBER = "0.3.0-0114-mt-tryout1"
AMETHYST_REPO = "https://git.novacow.ch/Nova/PyWebServer/"
@@ -143,7 +143,7 @@ class FileHandler:
def read_file(self, file_path, directory=None):
if "../" in file_path or "%" in file_path:
return 403, None
- if file_path == "api.py":
+ if file_path == "pes.py":
return 404, None
if directory is not None:
@@ -192,7 +192,7 @@ class RequestParser:
def extract_header(self, header: str, request: bytes | str):
if isinstance(request, bytes):
- request = request.decode("utf-8", "ignore")
+ request = request.decode("iso-8859-1", "ignore")
lines = request.splitlines()
for line in lines:
if line.startswith(header):
@@ -200,13 +200,13 @@ class RequestParser:
return value
return None
- def parse_request_line(self, line, host):
+ def parse_request_line(self, line, host, no_mod=False):
"""Parses the HTTP request line."""
try:
method, path, version = line.split(" ")
except ValueError:
return None, None, None
- if path.endswith("/") or ("." not in path):
+ if (path.endswith("/") or ("." not in path)) and (not no_mod):
if not path.endswith("/"):
path += "/"
index = self.file_handler.read_config("index", host) or "index.html"
@@ -276,6 +276,7 @@ class RequestParser:
class ProxyServer:
def __init__(self, fh):
self.file_handler: FileHandler = fh
+ self.rq: RequestParser = RequestParser()
@staticmethod
def recv_all(sock):
@@ -293,16 +294,24 @@ class ProxyServer:
def try_connection(
self, host: str, port: int, data: bytes, chost: str, force_tls: bool = None
):
- if port in [443, 8443, 9443]:
- do_tls = True
- else:
- if force_tls is True:
- do_tls = True
- else:
- do_tls = False
print(f"\n\n\nchost: {chost}\n\n\n")
nhost = self.file_handler.read_config("proxy", chost)
print(f"\n\n\nnhost: {nhost}\n\n\n")
+ # nhost will include http or https.
+ if nhost.startswith("https"):
+ nhost = nhost[6:-1]
+ do_tls = True
+ elif nhost.startswith("http"):
+ nhost = nhost[5:-1]
+ do_tls = False
+ else:
+ raise SyntaxError(
+ "Syntax error in config! Key: `proxy` Reason: `Expected http([...]) or https([...]), not "
+ f"{nhost[:6]}[...]{nhost[-1:]}!`"
+ )
+ if force_tls is True:
+ do_tls = True
+ print(f"\n\n\nnhost: {nhost}\n\n\n")
if ":" in nhost:
nport = int(nhost.split(":")[1])
nhost = nhost.split(":")[0]
@@ -311,47 +320,52 @@ class ProxyServer:
print(f"{nhost}, {nport}, {data}")
data = self.reset_host(nhost, nport, data)
try:
- return self.tcp_send(host, port, data, do_tls)
- except Exception:
- if do_tls is False:
- print("Retrying with TLS...")
- return self.try_connection(host, port, data, chost, True)
- else:
- raise
+ print("Waiting on TCP start.")
+ return self.tcp_send(nhost, nport, data, do_tls)
+ except Exception as e:
+ raise Exception(f"Server replied unexpected. Reply from Python subsystem: {e}")
@staticmethod
def reset_host(host: str, port: int, data: bytes):
- data = data.decode()
- data = data.splitlines()
- for line in data:
- print(line)
- if line.startswith("Host:"):
+ header_end = data.find(b"\r\n\r\n")
+
+ if header_end == -1:
+ return data
+
+ header_bytes = data[:header_end]
+ body = data[header_end + 4:]
+
+ headers = header_bytes.decode("iso-8859-1")
+
+ lines = headers.split("\r\n")
+ new_lines = []
+
+ for line in lines:
+ lower = line.lower()
+
+ if lower.startswith("host:"):
if port not in [80, 443]:
- new_line = f"Host: {host}:{port}"
+ line = f"Host: {host}:{port}"
else:
- new_line = f"Host: {host}"
- idx = data.index(line)
- data[idx] = new_line
- print(f"\n\n\n{idx}\n\n\n")
- if line.startswith("Connection:"):
- idx = data.index(line)
- new_line = "Connection: close"
- data[idx] = new_line
- data = "\r\n".join(data)
- data = f"{data}\r\n\r\n"
- print(data)
- return data.encode()
- # return data
+ line = f"Host: {host}"
+
+ elif lower.startswith("connection:"):
+ line = "Connection: close"
+
+ new_lines.append(line)
+
+ rebuilt_headers = "\r\n".join(new_lines).encode("iso-8859-1")
+
+ return rebuilt_headers + b"\r\n\r\n" + body
@staticmethod
def create_tls_context():
- # Create a context that by default verifies with system CAs
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return ctx
- def tcp_send(self, host, port, data: bytes, do_tls: bool):
+ def tcp_send(self, host, port, data: bytes, do_tls: booll):
try:
with socket.create_connection((host, port), timeout=10) as raw_sock:
raw_sock.settimeout(10)
@@ -363,11 +377,35 @@ class ProxyServer:
) as ssock:
ssock.sendall(data)
print("data reached")
- return self.recv_all(ssock)
+ resp = self.recv_all(ssock)
+ if self.rq.extract_header("Transfer-Encoding", resp) == "chunked":
+ ssock.sendall(b"TRANSER-ENCODING IS NOT SUPPORTED")
+ ssock.close()
+ resp = (
+ "HTTP/1.1 502 Gateway Error\r\nConnection: close\r\nContent-Length: 270\r\n"
+ f"Server: Amethyst/build-{AMETHYST_BUILD_NUMBER}\r\n\r\n"
+ "Gateway Error.\nThe upstream server tried to use a a transfer mode not "
+ "yet supported\nExact error:\nE_DATA_STREAMING_NOT_SUPPORTED\n"
+ "The unsupported mode:\nTransfer-Encoding: chunked.\n"
+ "There is no fix. The problem lies with the proxy, and is not a fault of the upstream server."
+ )
+ return resp
else:
+ print(f"\n\n\nraw data: {data}\n\n\n")
raw_sock.sendall(data)
+ print("Waiting for response...")
resp = self.recv_all(raw_sock)
- print(f"resp = {resp}")
+ if self.rq.extract_header("Transfer-Encoding", resp) is not None:
+ raw_sock.sendall(b"TRANSER-ENCODING IS NOT SUPPORTED")
+ raw_sock.close()
+ resp = (
+ "HTTP/1.1 502 Gateway Error\r\nConnection: close\r\nContent-Length: 270\r\n"
+ f"Server: Amethyst/build-{AMETHYST_BUILD_NUMBER}\r\n\r\n"
+ "Gateway Error.\nThe upstream server tried to use a a transfer mode not "
+ "yet supported\nExact error:\nE_DATA_STREAMING_NOT_SUPPORTED\n"
+ "The unsupported mode:\nTransfer-Encoding: chunked.\n"
+ "There is no fix. The problem lies with the proxy, and is not a fault of the upstream server."
+ )
return resp
except Exception:
raise
@@ -412,9 +450,11 @@ class WebServer:
self.http_socket = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
self.http_socket.bind(("::", self.http_port))
+ self.http_socket.settimeout(25)
self.https_socket_raw = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
self.https_socket_raw.bind(("::", self.https_port))
+ self.https_socket_raw.settimeout(25)
self.proxy_handler = ProxyServer(self.file_handler)
@@ -447,40 +487,39 @@ class WebServer:
self.running = True
def start(self, http, https):
- signal.signal(signal.SIGINT, self.shutdown)
- signal.signal(signal.SIGTERM, self.shutdown)
-
- http_thread = threading.Thread(target=self.start_http, daemon=True)
- https_thread = threading.Thread(target=self.start_https, daemon=True)
-
if https is True:
if self.skip_ssl is True:
print("WARN: You have enabled HTTPS without SSL!!")
yn = input("Is this intended behaviour? [y/N] ")
if yn.lower() == "n":
exit(1)
- https_thread.start()
+ self.start_https()
else:
self.https_socket.close()
if http is True:
- http_thread.start()
+ self.start_http()
else:
self.http_socket.close()
- http_thread.join()
- https_thread.join()
-
def start_http(self):
self.http_socket.listen(5)
print(f"HTTP server listening on port {self.http_port}...")
while self.running:
try:
conn, addr = self.http_socket.accept()
- self.handle_connection(conn, addr)
- except Exception as e:
- print(f"HTTP error: {e}")
+ if self.file_handler.read_config("threading") is True:
+ threading.Thread(
+ target=self.handle_connection,
+ args=(conn, addr),
+ daemon=True
+ ).start()
+ else:
+ self.handle_connection(conn, addr)
except OSError:
break
+ except Exception as e:
+ if not "timeout" in f"{e}":
+ print(f"HTTP error: {e}")
def start_https(self):
self.https_socket.listen(5)
@@ -488,13 +527,19 @@ class WebServer:
while self.running:
try:
conn, addr = self.https_socket.accept()
- self.handle_connection(conn, addr)
- except Exception as e:
- print(
- f"HTTPS error: {e}"
- ) # be ready for ssl errors if you use a self-sign!!
+ if self.file_handler.read_config("threading") is True:
+ threading.Thread(
+ target=self.handle_connection,
+ args=(conn, addr),
+ daemon=True
+ ).start()
+ else:
+ self.handle_connection(conn, addr)
except OSError:
break
+ except Exception as e:
+ if not "timeout" in f"{e}":
+ print(f"HTTPS error: {e}")
def handle_connection(self, conn, addr):
try:
@@ -528,22 +573,7 @@ class WebServer:
body += chunk
data += body
- #
- # print(f"\n\nbody length {len(body)}\n\n")
- # print("headers len", len(headers))
- # print("rest len", len(rest))
- # print("body len", len(body))
- # print("content_length", content_length)
- #
- # print("last 200 bytes of body:")
- # print(repr(body[-200:]))
- # print("body starts with:")
- # print(repr(body[:100]))
- # print(f"body: {body}")
- print(b"data: " + headers + b"\r\n\r\n" + body)
- # # data = conn.recv(32768)
- # print(f"len(data) = {len(data)}")
- request = data.decode(errors="ignore")
+ request = data.decode("iso-8859-1", errors="ignore")
if not data:
response = self.build_response(
400, "Bad Request"
@@ -617,14 +647,14 @@ class WebServer:
value = self.file_handler.read_config("proxy", host)
if ":" in value:
host = value.split(":")[0]
- port = int(value.split(":")[1])
+ port = int(value.split(":")[1][:-1])
else:
host = value
port = 443
return self.proxy_handler.try_connection(
host,
port,
- data.encode(),
+ data.encode("iso-8859-1"),
orig_host,
)
@@ -643,13 +673,25 @@ class WebServer:
or self.file_handler.base_dir
)
- if self.file_handler.read_config("apimode", host) is True:
+ if bool(self.file_handler.read_config("pesmode", host)) is True:
if not os.path.join(os.getcwd(), directory) in sys.path:
sys.path.append(f"{os.path.join(os.getcwd(), directory)}")
- import api
+ import pes
+ try:
+ pesclass = pes.PES()
+ threadcompat = pesclass.THREAD_SAFETY
+ # if not threadcompat:
+ return pesclass.on_request(data)
+ except Exception as e:
+ return self.build_response(
+ 500,
+ "Amethyst is currently unable to serve your request. Below is debug info.\r\n"
+ f"Error: {e}; Version: amethyst-b{AMETHYST_BUILD_NUMBER}\r\n"
+ "You cannot do anything at this time, the server owner has made a misconfiguration in their Python Extension Script",
+ )
- apiclass = api.API()
- return apiclass.on_request(data)
+ if bool(self.file_handler.read_config("621mode", host)) is True:
+ return self.build_response(621, "")
file_content, mimetype = self.file_handler.read_file(path, directory)
@@ -749,14 +791,14 @@ class WebServer:
if status_code == 621:
headers = (
- f"HTTP/1.1 {status_code} {status_message}\r\n"
+ "HTTP/1.1 302 UwU Nya!\r\n"
"Server: Amethyst/build-0621\r\n"
- "Content-Length: 30\r\n"
- f"Connection: close\r\n\r\n"
- )
- body = "https://e621.net/posts/6155664"
-
- print(f"{headers + body}")
+ "Content-Length: 0\r\n"
+ "Connection: close\r\n"
+ "Note: congrats, you found a funny. i guess.\r\n"
+ "Host: https://e621.net/posts/\r\n\r\n"
+ ).encode("iso-8859-1")
+ body = "".encode("iso-8859-1")
return headers + body
def shutdown(self, signum, frame):
diff --git a/certgen.py b/certgen.py
index 1a81cf1..f446e9c 100644
--- a/certgen.py
+++ b/certgen.py
@@ -6,28 +6,50 @@ import datetime
class AutoCertGen:
- def __init__(self):
- pass
+ def __init__(self, name="website", org="organization", locale="place", province="province", country="ZZ", dns="localhost"):
+ self.name = name
+ self.org = org
+ self.locale = locale
+ self.province = province
+ self.country = country
+ self.dns = dns
- def gen_cert(self):
- # Generate private key
+ def generate_self_signed_cert(self, different_issuer=False):
private_key = rsa.generate_private_key(
public_exponent=65537,
key_size=2048,
)
- # Define subject and issuer (self-signed)
- subject = issuer = x509.Name(
- [
- x509.NameAttribute(NameOID.COUNTRY_NAME, "ZZ"),
- x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "Some province"),
- x509.NameAttribute(NameOID.LOCALITY_NAME, "Some place"),
- x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Some org"),
- x509.NameAttribute(NameOID.COMMON_NAME, "localhost"),
- ]
- )
+ if different_issuer is True:
+ subject = x509.Name(
+ [
+ x509.NameAttribute(NameOID.COUNTRY_NAME, self.country),
+ x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, self.province),
+ x509.NameAttribute(NameOID.LOCALITY_NAME, self.locale),
+ x509.NameAttribute(NameOID.ORGANIZATION_NAME, self.org),
+ x509.NameAttribute(NameOID.COMMON_NAME, self.name),
+ ]
+ )
+ issuer = x509.Name(
+ [
+ x509.NameAttribute(NameOID.COUNTRY_NAME, "RU"),
+ x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "Красноярск Область"),
+ x509.NameAttribute(NameOID.LOCALITY_NAME, "Красноярск"),
+ x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Amethyst Group"),
+ x509.NameAttribute(NameOID.COMMON_NAME, "Amethyst Untrusted Signing Certificate"),
+ ]
+ )
+ else:
+ subject = issuer = x509.Name(
+ [
+ x509.NameAttribute(NameOID.COUNTRY_NAME, self.country),
+ x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, self.province),
+ x509.NameAttribute(NameOID.LOCALITY_NAME, self.locale),
+ x509.NameAttribute(NameOID.ORGANIZATION_NAME, self.org),
+ x509.NameAttribute(NameOID.COMMON_NAME, self.name),
+ ]
+ )
- # Create certificate
certificate = (
x509.CertificateBuilder()
.subject_name(subject)
@@ -39,12 +61,11 @@ class AutoCertGen:
datetime.datetime.utcnow() + datetime.timedelta(days=365)
) # 1 year validity
.add_extension(
- x509.SubjectAlternativeName([x509.DNSName("localhost")]), critical=False
+ x509.SubjectAlternativeName([x509.DNSName(self.dns)]), critical=False
)
.sign(private_key, hashes.SHA256())
)
- # Save private key
with open("key.pem", "wb") as f:
f.write(
private_key.private_bytes(
@@ -54,8 +75,5 @@ class AutoCertGen:
)
)
- # Save certificate
with open("cert.pem", "wb") as f:
f.write(certificate.public_bytes(serialization.Encoding.PEM))
-
- print("Self-signed certificate and private key generated for HTTPS server!")
diff --git a/html/index.html b/html/index.html
index 6af8333..229d6c7 100644
--- a/html/index.html
+++ b/html/index.html
@@ -8,7 +8,7 @@
Hello from Amethyst!
This page confirms Amethyst can read files from your PC or server and serve them to your browser!
This is a test page, if you aren't the server owner, they might not have finished setting up their site, be patient. If this doesn't go away after a while, tell them they've made an oopsie
- This server runs Amethyst build 0080
+ This server runs Amethyst build 0.3.0-0114-mt-tryout1