!!! Der Text wurde mit hilfe von Copilot erstellt als probe, viele teile habe ich aber ueberarbeitet. !!!

Wenn du einen Dedicated Server bei Hetzner mieten möchtest, dich Proxmox VE und IPv6 faszinieren und du Lust auf ein bisschen Tüfteln hast, bist du hier genau richtig. In diesem Blogartikel zeige ich dir, wie du einen Server mit Proxmox VE einrichtest und IPv6 im Routing-Modus mit OPNsense konfigurierst. Wir orientieren uns dabei an der Anleitung auf der Hetzner-Community-Seite: Hetzner Proxmox VE Tutorial.

Ich habe mir einen dedicated server nur mit ipv6 geholt, keine ipv4 dabei

Schritt 1: Installation von Proxmox VE

Um Proxmox bei Hetzner installieren zu koennen fragst du entweder per ticket ob sie dir ein USB stick mit an die Remote Konsole anschliessen oder du kannst theoretisch auch auf einem debian das von hand installieren ueber die repos.

Ich habe den ersten weg genommen.

Schritt 2: IPv6 und Präfix korrekt konfigurieren

Um sicherzustellen, dass dein Netzwerk und die OPNsense-VM reibungslos funktionieren, musst du das IPv6-Präfix korrekt konfigurieren.

Du musst dein prefix in vier elemente aufteilen:

1. Du brauchst eine statische IP fuer den Proxmox
2. Du brauchst eine statische IP fuer das interne VM Routing interface
3. Du brauchst eine statische IP fuer die opnsense
4. Ein Prefix fuer opnsense welches es verteilen kann innerhalb der prefix delegation von IP2

IP 1 muss sich in einem anderen prefix befinden als IP2 und IP3, und die letzten beiden muessen im selben Scope liegen als im selben Prefix. Das wird gleich klarer wenn wir zu den bridges kommen.

Ich habe mit Proxmox SDN und dnsmasq leider kein DHCPv6 ans laufen bekommen daher die OPNSense, sonst braeuchte man die auch nicht.

Schritt 3: Netzwerkkonfiguration im Routing-Modus

Jetzt geht's ans Eingemachte: Die Netzwerkkonfiguration.

• Öffne die Datei /etc/network/interfaces auf deinem Proxmox-Server und füge die folgenden Einträge hinzu:
• Stelle sicher, dass die IPv6-Adresse und das Gateway entsprechend der Hetzner-Anleitung konfiguriert sind.

Hetzner hat sein Gateway immer auf fe80::1 was aber durch das Router Advertisement kommt, standard IP fuer ipv6 Gateways.

Routing aktivieren: Um IPv6 im Routing-Modus zu betreiben, bearbeite die Datei /etc/sysctl.conf und füge folgende Zeile hinzu:

net.ipv6.conf.all.forwarding=1

Aktiviere dies mit sysctl -p

Host-Konfiguration:

auto vmbr0
iface vmbr0 inet6 static
        address 2a01:4f8:XXXX:XXXX::2/128
        gateway fe80::1
        bridge-ports enp0s31f6
        bridge-stp off
        bridge-fd 0
#Public Internet

auto vmbr1
iface vmbr1 inet6 static
        address 2a01:4f8:XXXX:XXXX:a::1/80
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up ip route add 2a01:4f8:XXXX:XXXX:a:a::/96 via 2a01:4f8:XXXX:XXXX:a::2
#Routed Subnet for OPNSense

Zur erklaerung:

Ein /128 spezifiziert nur eine einzige IP-Adresse. Daher reserviere ich für Proxmox selbst auf der Standard-Bridge vmbr0 diese IP zur Kommunikation via SSH, für Updates, etc., und verwende sie somit als Management-Interface. Die zweite Bridge ist das Routing-Interface. Hier setze ich ein a in den fünften Block, der Rest wird mit Nullen gefüllt. Als Präfix nutze ich die 80, wodurch alles in Block 6-8 unterhalb von a an die Bridge delegiert wird.

Das /64-Präfix ist der vollständige Präfix von Hetzner, der immer bis zum vierten Block reicht. Der Präfix entspricht der Maske wie bei IPv4 und teilt dem System mit, welcher Bereich unveränderlich zur Verbindung gehört und welche IPs das Netz nutzen kann.

Ich gehe also 16 Bits weiter, da ein Block 16 Bits entspricht—nur eben in hexadezimaler Darstellung. Ein Zeichen repräsentiert die Zahl eines binären Blocks von 0 bis f, was 16 Zeichen entspricht. F ist, wenn alle 4 Bits auf 1 stehen, in einem der 4 binären Bit-Blöcke, einfach die 4 in ihrer 16-Zeichen-Darstellung aufgeschrieben.

Und nach diesem exkurs gebe ich eine Route mit einem 96 Prefix an (64+16+16) und dort mit der gateway IP zu dem opnsense netz.

Schritt 3: OPNSense

Installiere nun eine opnsense VM in den proxmox und gebe ihm die vmbr1 als WAN und fuer LAN habe ich ein SDN vnet genommen.

in OPNSense konfigurierst du nun den routed 96 prefix und aber als Gateway die Bridge IP anstelle von fe80::1.

Wir schalten Router Advertisement ein!

Und konfigurieren den DHCPv6 und noch Unbound

Da ich nie mehr als 1000 VMs auf der kiste haben werde reicht es wenn ich einfach nur 4.096 ips auswaehle.

Unbound DNS hat auch eine kleine konfiguration

Ich habe neben den hetzner DNS servern von System fuer github einen sog. NAT64 DNS Server hinterlegt, weil github kann kein IPv6.

In this article you will see how you can use Ingest Actions in splunk to redirect data into S3-compatible storage. And how to retrieve it back from S3 into the splunk index.

Wait you might say, there is a AWS Add-On which is capable of ingesting from S3 you might say ? Yes and no….
It only is compatible to AWS S3 implementations, so the UI elements are restricted to AWS stuff.

So I had to develop my own. How ? HEC !

HEC ? … HTTP Event Collector

If you just want the script, go ahead, skip my walkthrough and learning path and get what you searched for, down below you can find the script

So now for the people who are willing to read throug my thought process.

Back story

A bit of a back story, you see if you do work with an API which is practical to implement into the REST API Modular Input from BaboonBones this is easy, but if you are not able to do so, you have to write you own script to get the data.

But you want it also in splunk, so how ? yeah you create an HTTP Event Collector, you can create basically an API token to post data right into the index queue and hope for the best.

No you have to format it in a specific way, ok you can just send the event field, but that is no fun.

What you really want to do is to send an event with all the original information you expect it to have, just when you would have collected it through the universal forwarder sooo:

• time
• source
• sourcetype
• host
• event

Now you have context.

So I knew that I can ingest data through an HEC, so I need to figure out how to connect to S3 and retrieve the data, and first check if this is at all usable data to work with, what splunk exports onto the S3 bucket.

But first lets get some data to go from the index queue out to the S3 bucket.

Configure Ingest actions to route data to S3 destination

Starting with the destination, you can configure it on the indexer manager, or on a search head connected, I assume you have the correct permissions going forward.

1. So you select Settings > Ingest Actions under DATA
2. Now you select Destinations
3. You Add a Destination
4. Now enter a unique name for the destination, I call it like s3-$sourcetype-$usecase
5. Enter your S3 Endpoint URL, just type it in, the Dropdown is not a restriction, it states it also there that you can enter anything
6. Enter your S3 Bucket, Folder (I set it to the name of the destination)
7. Enter your Access ID und Secret Key
8. Check with Test connection.

Please make sure under Advanced Options to disable Compression and enable the JSON format, so you can read it properly

So you now have a Destination.

Now go into Rulesets, click Add Rule

• Give it a name, I did also like s3-$sourcetype-$usecase
• Then select the sourcetype you would like to intercept
• Select a sample size to design the filter rule
• Click on Apply
• Then add a rule “Route to Destination”
• Select EVAL and create a rule like

source="WinEventLog:Security" AND match(_raw,"<EventID>4624</EventID>")

• Now remove “Default Destination”
• Select your specified new Destination for S3
• Your are done, now all Events matching that rule are now setup to be redirect to S3 and not ingested

Ingesting back into splunk

So now have a look onto S3 and view the files created.

As you see that out of luck your json files are just lists of event data formatted in a specific format.

Which format? when you read carefully you see that the data is right formatted like the object a HEC expects when posted onto it.

Sooooo lets begin and have fun!

// Btw. this looks easy now but it took a whole day to figure out

First I used python, because splunk is python and I may implement into a custom add-on.

So for working with with S3 on python AWS provides the boto3 module through pip, In my case I used conda but pip is fine.

I give you the complete list here because the rest are just modules you need anyway, no magic to unpack here

conda install boto3, requests

Connect to S3 with boto3

First you need to import boto3, duh. Then you connect an s3 resource and get your bucket selected

import boto3 
s3 = boto3.resource('s3', endpoint_url='https://s3.example.com', aws_access_key_id='ADBCDEF', aws_secret_access_key='GHIJKLMNOP') 
bucket = s3.Bucket('YOURBUCKET')

You remember I have saved my data into a folder, but s3 does not understand folder, it understands prefixes .. path prefixes. So now on to getting you files from s3 loaded into python.

There is a object called objects on the Bucket object, it understands the .all() method and the .filter() method. to filter for any information we need yes, the filter method and our prefix.

bucket.objects.filter(Prefix='s3-$sourcetype-$usecase')

This will give you every file in that prefix. We simply need to foreach it retrieve the json file and send it onto splunk, easy!

You might want to stop here, unfortunately boto3 or the S3 specification does not just stream you data into a variable, you need to specifically download it, but not save it, here is the catch ! SpooledTemporaryFile a function inside of tempfile. To use it you need to import tempfile and right now you can also import json and requests.

You need to specify how many bytes you are allowing it to save into memory before falling back to disk to save it, and that is 1MB, since I saw that the file are not really get bigger than that.

import requests 
import tempfile 
import json 
#[...] 
for object in bucket.objects.filter(Prefix='s3-$sourcetype-$usecase'): 
  data = tempfile.SpooledTemporaryFile(max_size=1000000)

Now you have a place to save the data into memory, to now retrieve it you need one of the two functions, download_file or download_fileobj.

download_file expect you to save it into a path, we do not want that.
download_fileobj does exactly what we want, it downloads the file as an object into an object.

bucket.download_fileobj(object.key, data)

Now hooray we have the data, so we see that the data is json, so we load it and foreach over the items in the json list.

We know that we can just post it onto the HEC, so also lets do that. Keep in mind that we need to define the authorization header for splunk

headers = {'Authorization': 'Splunk dd4680f4-bd33-426e-9ba6-f7ceff5761cd'} 
events = json.loads(data.read()) 
for event in events:  
  requests.post(headers=headers, url='https://splunk.example.com', json=event)

If we now execute that we see….
Nothing…

As I found out after working through the SpooledTemporaryFile docs I found that since it is a datastream I need to set the handler back to the front, because we wrote into it it was at the back with no data behind the last byte.

We do that with:

data.seek(0)

Thats it, now it works, let us check

Eureka, it works, now we can import that data into splunk as we need it, so we can redirect data not needed away into s3 and retrieve it when we need it.

Soo there is still something more to keep in mind. When now ingesting through the HEC, the event just looks like it comes from the original source, because the source is in the event data and we have not touched it.

moving forward you will see that it lands directly back again in s3, so we need to differentiate between data coming from the original source and the HEC.

So I do this, do identify the data and the file and bucket and so on it came from:

event['source'] = object.key

The key by the way is the filename of the object you currently have retrieved.

To now finally have the complete script you can go ahead

import boto3 
import requests 
import tempfile 
import json 
 
s3_endpoint = 's3.example.com' 
s3_bucket = 'bucket' 
s3_prefix = 's3-xmlwineventlog-event4568' 
s3_access_key = 'ABDEFG' 
s3_secret_key = 'HIJKLMNOP' 
 
splunk_server = 'splunk.example.com' 
splunk_token = 'dd4680f4-bd33-426e-9ba6-f7ceff5761cd' 
 
headers = {'Authorization': 'Splunk ' + splunk_token} 
 
s3 = boto3.resource('s3', endpoint_url=s3_endpoint, aws_access_key_id=s3_access_key, aws_secret_access_key=s3_secret_key) 
bucket = s3.Bucket(s3_bucket) 
 
for object in bucket.objects.filter(Prefix=s3_prefix): 
  data = tempfile.SpooledTemporaryFile(max_size=1000000) 
  data.seek(0) 
  bucket.download_fileobj(object.key, data) 
  events = json.loads(data.read()) 
 
  data.close() 
 
  for event in events:  
    event['source'] = object.key 
    requests.post(headers=headers, url=splunk_server, json=event)

In the next post on this topic I will go ahead and will show an optimized version of a splunk modular input

When configuring AppLocker there are several things to think about. AppLocker itself a tool from Microsoft integrated into Windows to allow Administrators and also Microsoft itself to block applications based on three different identifiers.

The Identifiers are:

• Path
• Hash
• Publisher

Microsoft does configure AppLocker with some Default rules if you wish to enable them.

The requirements to run AppLocker is at least Windows Pro edition.

Rules are enforced as a default, so if you apply rules then they are enforced, no rules mean nothing to enforce. So to first verify your rules please first set the AppLocker to Audit mode, this is done in the same place were you configure the rules itself.

Path

The Path rule checks for the path of an application, this can be easily circumvented if you can move or install an application outside of this specified path or filename.

Hash

A Hash rule is exactly meant to just verify the hash of an application to be blocked and nothing else, this is the most direct and specific rule options. But you need to be aware that this is somewhat work intensive because you need to keep the hashes up to date if they ever change.

Publisher

The Publisher rule is the most powerful in my opinion.

You can filter for a specific publisher based on the vendors Code Signing certificate for an application. But you can also make use of wildcards, so you can allow or deny specific application vendors, products, executables and down to the version or ranges of versions.

The AppLocker „Control Panel“

AppLocker policies are configured in three locations but basically these are just two locations.

• Domain Group Policy Management
• Local Group Policy
• Local Security Policies

The Local Group Policy and Security Policy do sync each other though. These Screenshots show you a brief overview of the options in the panel

Hidden Options

A hidden option which is hidden for a reason is the DLL Rule Collections.
This will degrade your systems performance and will audit every DLL to be loaded for an AppLocker Allow/Deny rule.

AppLocker Processing Rules

• A deny rule always overwrites an allow rule
• If a rule exists even if it is an allow rule is a implicit deny for everything else
• If a Exe Rule exists than all packaged Apps are blocked also until you allow the packaged app or allow every packaged app

AppLocker Event Log

You can find the Event Log for AppLocker in the Event Viewer at: Applications and Services - Microsoft - Windows AppLocker

And then you have a Log per Category in AppLocker

Use Case Examples

Blocking a specific group to execute an application

So you want to block a specific group to execute an application than you create a deny rule for the application targeted to the group, this is one of the easier things to do in AppLocker.

Allowing a specific group to execute an application

For example you want to deny users to execute well known browsers because you have a managed browser for your environment. Then you need to think out of the box because you can‘t just Deny the application and than allow it to a group, because a deny wins over an allow.

So how you do it ?

1. You create a rule for Everyone to Allow Anything, this is done with a Path rule and the Path to be the wildcard symbol „*“.
2. You then go into the Rule and under Exceptions place every Application you want to block, this works because you make an exception to an allow which then is implicitly denied.
3. Now you create a Allow rule for the exact applications you just made an exception for in the Allow Everyone Anything rule and assign it to the group you wish to attach it to

Blocking vulnerable Applications

Here we make use of the Publisher rule, you can basically set the rule to allow anything of this Publisher and Application but deny access to applications with a version number below a specific point, combined with hash rules you have a pretty powerful Method of blocking vulnerable applications for your users.

Hi,

My Nextcloud instance broke a couple of days ago and could’nt get it running again on the original config, to recover my contacts I looked in the database and I find the raw vcard data (which is the common sharing format of contacts) in the database.

With this code in python you can restore the files for you to import them to any addressbook like Google, Outlook or iCloud, or back into a Nextcloud.

please note that you need to search for your addressbook Id via SQL before. You could use adminer or a graphical SQL-client.

Hi, ich bin gerade dabei mein Offsite backup vom S3 Storage Hoster Wasabi zu eine Hetzner StorageBox umzuziehen.

Leider bietet mir Hetzner kein S3 Protokoll an um die StorageBox anzusteuern, ich kann meine Backups auch via rsync auf die StorageBox sichern jedoch wollte ich ausprobieren ob das nicht auch mit S3 moeglich waere.

Nun ich bin auf MinIO gestossen, eine Open Source Implementierung des S3 Storage Protokolls.

Da ich bereits einen Hetzner vServer habe als Ingress Point zu meinen Diensten habe ich mir gedacht, koennte ich doch solch einen nehmen um meine StorageBox via S3 zugaenglich zu machen.

Storagebox einbinden

Zu aller erst habe ich in der StorageBox SAMBA aktiviert und externen Zugriff deaktiviert, damit ich mir um die SMB Credentials uebers Internet nicht soviel sorgen machen muss.

Nun auf dem vServer habe ich dann die cifs-utils installiert und dann den cifs mount in der fstab eingetragen und die StorageBox zu eingebunden.//u294301.your-storagebox.de/backup /mnt/u294301 cifs defaults,auto,uid=1000,gid=1000,user=u294301,password=***********

MinIO konfigurieren

MinIO verwendet fuer seinen standard dienst den user minio-user welcher noch angelegt werden muss, da es sich bei mir um einen blanken rockylinux server handelt, denn ich bei bedarf direkt mit root verwalte habe ich so auch keinen anderen user gehabt, weshalb bei mir dieser die id 1000 erhalten hat, welche ich auch angegeben habe beim mount.

um MinIO nun zu installieren erstelle ich zuerst den user mit:$ adduser minio-user

dann holfe ich mir den aktuellen RPM installationsbefehl von der minio download page: https://min.io/download#/linux und waehle RPM aus$ dnf install https://dl.min.io/server/minio/release/linux-amd64/minio-20220308222851.0.0.x86_64.rpm

Dies hier oben ist die aktuelle Version von Heute. Nachdem die installation durch ist muss man noch die minio config erstellen, unter /etc/default/minio/etc/default/minio:
MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=**************
MINIO_VOLUMES="/mnt/u294301"
MINIO_OPTS="--console-address :9001"

Damit wird der admin account angelegt, das volume definiert und die Console eingeschaltet, um den S3 Dienst zu verwalten.

In diesem Fall ist aber noch kein SSL eingerichtet, wenn man jetzt mit$ systemctl enable --now minio

den Dienst starten wuerde.

SSL einrichten

Folgende einstellung ist noch in der config file notwendig:MINIO_SERVER_URL="https://s3.storagebox.mydomain.de"

Nun muessen wir auch dieses SSL Zertifikat erstellen, dafuer gibt es auch eine anleitung fuer LetsEncrypt: https://docs.min.io/docs/generate-let-s-encypt-certificate-using-concert-for-minio.html

In meinem fall werde ich dies manuell mit certbot machen. Der Ordner wurde dafür bereits von MinIO angelegt. Dieser ist unter /home/minio-user/.minio/certs zu finden.

Zum anlegen des LetsEncrypt certificates ist folgendes notwendig einzugeben:$ certbot certonly --standalone -d s3.storagebox.mydomain.de --stable-oscp -m hostmaster@mydomain.de --agree-tos

Nun nachdem die Erstellung erfolgreich war kann ich nun das zertifikat an den richtigen Ort kopieren

$ cp /etc/letsencrypt/live/s3.storagebox.mydomain.de/fullchain.pem /home/minio-user/.minio/certs/public.crt 
$ cp /etc/letsencrypt/live/s3.storagebox.mydomain.de/privkey.pem /home/minio-user/.minio/certs/private.key 
$ sudo chown minio-user:minio-user /home/minio-user/.minio/certs/private.key 
$ sudo chown mini-user:mino-user /home/minio-user/.minio/certs/public.crt

Hier ist es wichtig dann auch den certbot-autorenew zu aktivieren und die kommandos von oben in ein post-hook script zu speichern damit bei jeder erneuerung auch minio die zertifikate am richtigen ordner hat.

Region konfigurieren

Man hat auch noch die Moeglichkeit die Region zu konfigurieren mitMINIO_SITE_NAME="fsn1-dc14"
MINIO_SITE_REGION="eu-central"

GANZ WICHTIG!!!: bitte stelle den root account von der Console auf ein sehr sicheres passwort was auch mehrmals rotiert werden sollte. oder im privaten gebrauch blockiere per firewall den Zugang auf die Konsole wenn sie nicht gerade benoetigt wird.

Hi,

I’m Piere, I manage some standalone windows systems for friends and family members as VMs for Office use and some other stuff.

I don’t want to manage them through a Domain because using a DC and VPNs and remote systems is so much work.

Therefore I seeked some simpler solution for this.

I noticed that the Windows gpclient (Group Policy Agent) stores it’s local group policy at the following location: C:\Windows\system32\GroupPolicy. And I know a software to synchronize those settings around to other systems regardless of their network connectivity: syncthing.

To access those folders the process needs to be system.

So I did the following:

1. Download syncthing for Windows (https://syncthing.net/downloads/).
2. Extract to to C:\Windows\System32\config\systemprofile\syncthing
3. Now add this executable as a Task to the Task Scheduler

Establish Group Policy synchronization

I created for this a folder in the Task Scheduler library named IT Management

I changed the User account the task is running at to system and set it to ignore power settings (conditions) and to do not stop.

And I started the Task, here you can get the task config file you could import:
After this the SyncThing WebUI is accessible at http://localhost:8384 After logging in I declined the dialog to send usage data and deleted the default Sync folder. You get asked to not start syncthing as a system, for our use-case this is OK because this is actually intentional. To protect the settings I setup to protect the Web UI with a Admin Account.

This is for the Main system where the settings are configured and pushed out from. On the Client systems this is not necessary but for management purposes you could set the GUI Listen Address to some ZeroTier Admin Network and restrict in ZeroTier the Admin PC to connect to the IPs. So you could setup other folder syncs if you wish to remotely.

Now after I configured all the systems with SyncThing and ZeroTier (only if you wish too) I add the to the main system as trusted hosts.

Now I add the new folder to sync to all the clients and use the folder path for the GPOs I mentioned earlier and a proper Folder ID like (Group Policies). and a Folder Type of Send Only and Ignore Permissions.

Now all Client’s get a push notification to Add the System to their trust list and to add the shared folder, which you accept and use the same folder and set this to Receive Only and Ignore Permissions.

I also have some Registry Configurations I set with this method, the Local Group Policy editor does not allow to set these. therefore I set a PowerShell Script to set these instead. Is just as easy as with the GUI version, potentially easier.

Please be aware to set HKCU Settings as a User Logon Script rather than a Computer Startup Script.

If you click on Show Files you can see the folder where those scripts are used from, there you place yours.

Side-Note: to deploy applications I use winget and choco (chocolatey) as powershell startup scripts.

To get reports and inventories from the client I sync back a folder at the system account which contains reports generated from powershell scripts running periodically setup as a Task through a PowerShell Script.

Working with Firewalls and Microsoft Products it is sometimes necessary to allow certain endpoints in your firewall. Unfortunately not all are available through their provided Azure IP and Office 365 IP lists.

• https://www.microsoft.com/en-us/download/details.aspx?id=56519
• https://docs.microsoft.com/de-de/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide

They are only available as either a list in docs.microsoft.com or as an excel file, those are formats I can not use with my firewall.

To resolve this issue I build a AzureApp which polls these sources and create a json file which is consumable by many indicator-aggregators, like minemeld from Palo Alto Networks.

Defender ATP

When using Defender ATP there are certain Domains and IPs to be available to every system, regardless of their internet access policy, microsoft provides a excel file to allow you to view those endpoints: https://download.microsoft.com/download/8/a/5/8a51eee5-cd02-431c-9d78-a58b7f77c070/mde-urls.xlsx

Intune

The same is with Microsoft Intune for Autopilot or management in general. To allow for a stable connection they provide a list in their docs: https://docs.microsoft.com/en-us/mem/intune/fundamentals/intune-endpoints

The Azure App

I wanted to allow everybody to just consume those feeds, without having to setup a parser on their side, if you want to, it is open-source available here: https://github.com/pierew/serviceendpoints.azurewebsites.net

I had to first get off by understanding how to publish my docker container in Azure App Service, you can read that here: INSERT LINK

Then I build the framework for the container, an endpoint file, necessary resources and packages for alpine were easy to find, which wasn’t easy to get the correct converter for my file formats being present.

After some searching I settled on these two python packages:

• xlsx2csv (pip)
• mdtable2csv (https://github.com/tomroy/mdtable2csv)

those commands are just as simple as they sound, you pass in the file you want to convert and they place the conversions in the current working directory or beside the specified file.

After I had the excel file to converted into some csv files I could parse them in bash (yes I know, I could do all of that in python but I wasn’t this dedicated then). I first had to make sure that the service categories where in a machine readable form, therefore I searched and substituted these through a case-switch loop

This was the first version of the service, then available as intelligencefeeds, I changed the name now to serviceendpoints.

After that the need for Intune came up, so I searched for these resources, they are in a docs page, goosh, I thought I had to parse html but no! Microsoft has their docs in github as markdown files. Where Markdown files are there is a possibility to convert them to something else, I was right.

I extracted the table from the Markdown and used named package to convert them into csv files. which I used to create plain text files to serve for now.cat ./intune-endpoints.md | awk '/client accesses/,/## Network requirements/' | grep -v "client accesses" | grep -v "## Network requirements" > ./intune-management-endpoints.md

My next problem was to sanitize the output from comma,spaces,lines and brake lines into new lines, sed come in handy for that:tr " " "\n" | sed '/^$/d'

Now the plain text generator was done, but I wanted also a json for minemeld, so I went back to the code editor and fiddled around. I knew jq was the tool to do json stuff in the command line. I quickly got a jq command working which created me proper json for every item I wanted to have in the json file.

Basically for every indicator aggregator each json object is one indicator, each domain, url, ip is his own indicator.jq -n --arg type "URL" --arg category "management" --arg url "$item" '{type: $type, category: $category , url: $url}'

These json elements still needed to be formatted to fit the json frame around it, so I used sed again:| sed 's/}/},/' | sed 's/^/    /' >>

Not all IPs were listed for their domains, so I had to implement a dns lookup which looks like this:for ip in $(host -t a $item | grep "has address" | cut -d" " -f4)

now the complete logic for the App was done, I published it now under it’s new name at: https://serviceendpoints.azurewebsites.net

Minemeld

Minemeld is a indicator aggregator to prepare indicator feeds for primarily Palo Alto Networks Firewalls, Minemeld is available as a docker container from PANs dockerhub profile “paloaltonetworks/minemeld”

You can run the container with:docker run -dit --name minemeld --tmpfs /run -v minemeld-local:/opt/minemeld/local -v minemeld-logs:/opt/minemeld/log -p 8443:443 -p 8080:80 paloaltonetworks/minemeld

To now have minemeld query those feeds you need to create new prototypes based on existing one’s, you do that by navigating into the config and opening the Prototype Browser which is the blue “Hamburger” Button.

Now you need to have an existing Prototype to create a new one from to modify it, it needs to have the correct parent-class to parse JSON. Fortunately aws.S3 is a json prototype, select it and click NEW.

Now use the miner configs from my GitHub: https://github.com/pierew/minemeld

After creating this new prototype, select it again in the browser and “clone” it, this creates the NODE which will pull the feed and filter it based on the set attributes and filters.

Now go back into the prototypes and clone the feedHCGreen Output, which will be responsible for providing the feed for your Firewall to be read properly.

If you want you can use an aggregator prototype for IPv4 and URL to aggregate those MS products into one feed, so you need only one feed for every indicator type, if you want to.

you can also place those prototypes into the /opt/minemeld/local/prototypes/minemeldlocal.yml file and it will be autodetected.

If you wish to have these configured for you, you can use my docker container here: https://github.com/pierew/minemeld-docker/pkgs/container/minemeld-docker

Photo by Compare Fibre / Unsplash

Hi i tried to publish the syncthing WebGUIs from my DMZ systems to my internaly accessible haproxy VIP on my pfSense firewall and couldn’t figure out WHY I can’t connect to the service, it seems to have error “503 Server not found” until I choose to use the default backend.

I was stuck for half an hour, this moment as I write this blog entry. I figured out:

HAProxy refers to the first match of the acl per IP in the frontends, NOT WITH THE PORTs in mind. I had to use a different ACL check that matches only this frontend I wanted.

Originally published at https://www.pierewoehl.de on September 18, 2021.

Photo by Roman Synkevych / Unsplash

I had the need to connect my GHCR to an Azure App Service for CI/CD, but I didn’t want to use GitHub Actions as Microsoft would like to have the integration done, I didn’t like their implementation.

How I’ve done it ?

See, you can use a private registry with Azure App Service

the Image is public, therefore I don’t need any credentials passed in. To have the Continuous Deployment part enabled I created a GitHub Action curling the webhook url, since GitHub repository webhooks don’t accept in-line authentication as Azure provides the url this way.

Here are the GitHub Actions I created for you to re-use:

Just make sure to update the URL for azure, you need to use your endpoint which is the “name of the service” in my case it is intelligencefeeds.

If you’re curious this is a service providing the defender-atp service endpoints from microsoft and others following to use in your firewall

Originally published at https://www.pierewoehl.de on September 18, 2021.

I have quite a complicated UniFi Networking Setup segregating over 20 different networks and was using a well-known Third-Party UniFi Hosting Provider since 2018.

3 days ago they had updated their UniFi Controller, which I was using for free as a long-term Beta Member from years ago, it is broken, they did not admit it neither offered to help.

I requested to provide me with a backup since they have cut the Site Migrate and Backup functionality out of my account, they refused, so I was in need to re-do my network…

Since my current config was still running on the UniFi Gear I thought about a smooth transition instead of rebuilding from scratch. Since I didn’t want to completely overhaul my Rack just to have my proxmox node delivering the unifi controller as a VM be always connected to the gear and mess up my HA setup for it.

I have my UniFi Controller published for control traffic to serve other locations I support (Friends, Family) so I had to make sure it was always available from the Internet, this would also make my migration process easier, since the gear only needs internet to be adopted and re-integrated into the network

The Before

Above you see the basic Map view of the network and after that the Network Setup seen from a broader look.

The Plan

1. Install the new UniFi Controller into a VM on one proxmox host.
2. pre-configure the vlans and networks from documentation
3. Publish the Controller via pfSense/HAProxy
4. changing unifi controller dhcp ip in usg via cmdline
5. reset Access Points one by one
6. adopt them by the new controller
7. reset the switch2 and adopt it
8. prepare USG ports on switch2
9. move USG Patch cables (one by one keeping uplink)
10. Reset Coreswitch (activating the STP blocked secondary link between switch1,switch2)
11. adopt coreswitch and configure for DSL modem, USG und trunk ports
12. move proxmox node and one AP to switch2
13. reset switch 1, adopt it and configure NAS,Proxmox,UAP ports
14. move proxmox and AP back

The Result

All went smoothly because I had planned for redundancy of internet, proxmox uplink und switch links so I never had to connect via ethernet to any switch to configure something being isolated from the rest of the network.

Well no, I had one issue to overcome, when resetting the DSL switch I lost connectivity to the unifi controller vm, since it’s uplink was to the DSL connected pfsense, so I moved the VM into the VLAN of the modem to directly connect it the the Provider-NAT, just for the adoption of the core-switch.

But wait where is the USG ?
So this was a bit trickier, I had to make sure the controller had internet access while the central router was down, how I made it work ?

You remember the LTE Uplink being available to pfSense as a Fallback for the published services ?
Well here in Germany our LTE providers don’t give us public IPs with Port Forwarding, while the contract was already a Home Internet Access not just a Data plan.

So I have a cloud hosted VM being a VPN exit node for the LTE connection, I simply added my DSL pfSense via VPN to the same instance and configured NAT to failover to each of the links when one goes down, effectively bypassing my CloudFlare Load-balancer listening on Port Forwarding on the DSL.

With this setup I could reset the USG while the controller was connected trough the VPN connections bypassing the USG

How I connected to it after the reset?
I created a WLAN to be connected to the USGs LAN Native Network where I connected with my Mac to configure It.

After the USG was adopted I managed to migrate my entire unifi network to a new controller without any interruption in any hosted service.

The side-effect I can now cancel my CloudFlare Load-Balancer subscription!

Originally published at https://www.pierewoehl.de on June 12, 2021.

Hi Community, I was setting up my pfSense to proxy through web resources so I didn’t have to deal with updating firewall rules when service change and allow internet and dns in general to the containers and vms in general.

To do this you have to do some rewrite rules in the backend server definition

Let’s split this rule up into its parts to understand it a bit more.

• The “%[]” indicates a function to run, or more like “use this input in parameter one and process it through function defined in parameter two”
• The “path” is the variable containing the current path used by the request
• The “regsub” is the function using regex with function “s” to substitute input via regular expression
• The Input expression for the substitute “/linux/epel/?” is the incoming path from the frontend to be matched, the “?” says everything after that is passed through
• The Output expression for the substitute “/pub/epel/” is the path to rewrite to, so for example.

Request on Frontend: mirror.woehl.network/linux/epel/ Rewrite on Frontend to Backend Server: [dl.fedoraproject.org]/linux/epel/ Rewrite on Backend to new Path: dl.fedoraproject.org/[pub/epel/]

Originally published at https://www.pierewoehl.de on December 18, 2020.

Today I planned to update my wordpress alpine lxc template, So I cloned it into a temp VM, edited and updated the VM and wanted to delete the old template and overwrite it with the new, dang… it doesn’t work

Yep I had link-cloned it for another project container, sooo I had to do it the hard way.

How you ask ?

So Proxmox is running on ZFS as the storage driver, to resolve this issue I had to undo all linked references to the parent disk, or zfs volume if you are specific.

So I done it before but had to think twice how to go with it.

You do the following:

1. Go through your VMs/LXC Container and look for a disk reference with a basedisk-xxx reference in it, this shows the reference to the basedisk of your template.
2. Select the Disk
3. Click on Move Disk
4. Select the storage your current disk lies on (no worries, it will move(clone) to a local disk as a full clone)
5. Change the view to the snapshots.
6. Delete all snapshots from the past.
7. Now select the disk from the Resources/Hardware overview and delete the now unused disk with basedisk-xxx/subxxxx

Now you can delete your template and replace it with the new.

Originally published at https://www.pierewoehl.de on December 4, 2020.