I’m not sure how great a need there is for another proxy to circumvent the Great Fire Wall (GFW) for expats living in China. You’ve probably done your homework and obtained a VPN before coming here, or have some VPN service already available from your company. Nevertheless, seeing that the government is increasingly tightening its grip on the Internet and killing a few previously functioning paid VPN services, and that there seem to be few posts about Shadowsocks out there available in English, I feel it wouldn’t hurt to contribute another entry here. The advantage of such a proxy is obvious: since you’ll be the one who operates your own server with presumably limited bandwidth usage, it would be very hard for the authority to find it out, target it and kill it off. It has become a kind of de facto GFW circumvention tool for many programmers. Hopefully this post would be of some use to you and help you access the Internet more happily in China, or whatever place with heavy Internet censorship. At least this would also serve as a note for myself if I were to perform a new installation some time later.
A word of caution before I proceed: Shadowsocks refers to a protocol and a family of open-source software which implement the protocol, originally conceived by one developer (@clowwindy). Its encryption relies on a pre-shared key and block cipher algorithms. It focuses on being lightweight and fast, however the security aspects of the Shadowsocks protocol likely haven’t been rigorously evaluated by security experts. It’s good for daily needs, but you probably won’t want to solely rely on it if you need total anonymity/a particularly strong level of protection.
Spinning Up a VPS (Virtual Private Server)
Shadowsocks is a proxy software, it means you’ll first need a server to proxy the traffic with. There are a handful of decent VPS (Virtual Private Server) providers out there with attractive pricing (starting from $5 per month), for example:
(You may notice that referral codes are included in some of the links. Signing up from a referral can get you $10 credit at DigitalOcean, while Vultr offers $5 credit currently).
There are also some other low-end options available at lower prices. However they don’t usually offer servers in Asia:
If you’re in China, it would make sense to choose a server in the Asia-Pacific region. Vultr offers Tokyo and Sydney, while DigitalOcean offers Singapore and most recently, Bangalore. The Tokyo and Singapore servers are reported to work pretty well. However, if your network, like mine, is provided by China Telecom, it seems that the Bangalore/Sydney server’s performance far exceeds those of the previous two. I was only able to watch YouTube in 240p with the Tokyo server, but now 720p isn’t much of an issue. Or maybe it’s just because the region only has relatively few users by now. In any case, YMMV and it’s always best to try them out by yourself, either via the official speedtest or better, just set up the proxy and test it in action: The servers are billed per hour so just performing some tests won’t rake up huge costs.
I’ll demonstrate using the example of a server running Ubuntu 16.04 LTS on DigitalOcean. This process should be applicable to other VPS providers as well since they all provide Ubuntu as an option.
As shown in the picture below, Choose Ubuntu 16.04 LTS x32, $5/mo option, and the appropriate region.
Set up your user account. There is an excellent tutorial series on DigitalOcean detailing the whole process:
Step 2, add a new user. This is actually not mandatory for setting up Shadowsocks but it’s a part of the common procedure of setting up a Ubuntu server. It should make the system more secure.
For the record, the following are all the commands needed to add a user called “demo”. The $local prefix means the command is executed on your local machine:
local$ ssh root@SERVER_IP_ADDRESS
gpasswd -a demo sudo
local$ ssh-copy-id demo@SERVER_IP_ADDRESS # Optional, add public key authentication
vim /etc/ssh/sshd_config # Change PermitRootLogin to no. You may also use nano instead of vim
service ssh restart
Install the server application for Shadowsocks
After you’re able to log in as the new user you’ve just created (via ssh demo@SERVER_IP_ADDRESS), you can now install the server application for Shadowsocks. There are a few implementations in different languages available. I’m using the Go version by @cyfdecyf since
It’s easy to run. Just download the binary and you’re good to go.
It supports multiple clients with a single process. This means I can simultaneously let my computer and my phone use the proxy from the same server without further complication.
mkdir shadowsocks && cd shadowsocks to make a directory for shadowsocks.
Find the latest release of Shadowsocks and download the archive. To download the currently newest release for 32-bit Linux, run curl -L -O https://github.com/shadowsocks/shadowsocks-go/releases/download/1.1.5/shadowsocks-server-linux32-1.1.5.gz
Extract the binary by running gzip -d shadowsocks-server-linux32-1.1.5.gz
Make the binary executable by chmod a+x shadowsocks-server-linux32-1.1.5
Make a configuration file for the server by executing vim config.json (or nano config.json) and pasting in the content. The following is an example. Note how in this example I specified two separate ports for computer and cellphone, with different passwords “computer” and “mobile”. The full documentation of the file format is available at Github
Run the server by executing ./shadowsocks-server-linux32-1.1.5 You should see the server displaying startup messages. We’re done here.
(Should there be any abnormality on the server, you can just restart the VPS (via the command line or using control panel of DigitalOcean), ssh into the VPS by ssh demo@SERVER_IP_ADDRESS, and start this application again by cd shadowsocks && ./shadowsocks-server-linux32-1.1.5)
Install the client
The full range of clients available is documented at the official website. Choose the one that suits your platform. I recommend the Qt5 versions for their intuitive GUI. Besides, the alternative version shadowsocks-win didn’t seem to work for me on Windows and is only available in Chinese. The following are the steps for configuration on Ubuntu desktop. The process should be similar for any other system, including Windows and Android.
install the app from the corresponding ppa (on Windows you would just directly download and open the .exe file):
Launch the app from command line. Note that although the package is called shadowsocks-qt5, the binary itself is called ss-qt5!
Create a new profile. The following is my configuration file. Please fill in your own server address and password. Ensure your encryption method matches that specified on the server.
Important: Remember to select the option “One-time authentication”
If the connection is successful, then “test latency” in the client app should give you a reasonable latency figure.
Configure proxy settings. There are some methods to set the global proxy for the system, but the more common use case is to configure on an app-by-app basis, which would also be more reliable and flexible.
For Firefox, the setting is at Preferences -> Advanced -> Connection.
Important: Remember to tick the option “Remote DNS”. Otherwise the DNS poisoning attack will still be in effect.
For the stable version of Chrome, there is no built-in setting for proxy. A plugin called SwitchyOmega will help. Since you probably can’t reach Chrome Web Store without circumventing the GFW, you’ll have to search for and install the plugin manually. You may directly download it here. The blogger auooo has also made it available on a web storage. Drag the file into the Chrome “extensions” page (enter chrome://extensions/ in your Chrome address bar to access the page) to install it. Once installed, the configuration is as follows:
Also remember to select the shadowsocks profile by clicking on the SwitchyOmega icon on the menu bar:
For Dropbox, the setting is directly visible among the configuration tabs.
You should be able to apply the similar configuration procedure for other apps that need proxying.
The Android app should be able to proxy all traffic globally by default without the need of manually configuring proxy.
For those who can read Chinese, there is a very detailed post by auooo with step-by-step screenshots of client configuration on every operating system. You may want to refer to that post as well.
There is a bug in Ubuntu 16.04 LTS’s default fallback CJK font package fonts-noto-cjk that makes Chrome/Chromium render “thin” weight of the font for Chinese/Japanese/Korean characters, whenever the page’s language is not explicitly set to cn/jp/ko. This makes the characters unreadable. For example, in a Google search page which contains both English and CJK texts, the CJK text will be rendered thin.
I’ve reported this bug. It seems the issue is caused by the fact that fonts-noto-cjk uses an all-in-one file which contains all the weights of the font. The problem probably lies with fontconfig or Chrome/Chromium’s font fallback mechanism.
In the meantime, the temporary fix is to remove the package, download the font Noto Sans CJKfrom Google with each weight contained in an individual file, and manually install the fonts. Then Chrome/Chromium should use the correct weight of the font to render CJK characters.
A member of the Ubuntu Development Team, Gunnar Hjalmarsson, has made a personal PPA containing the font files individually. You may also consider directly using that PPA.
Org mode is already a part of my life. It’s only natural I’d like to perform various customizations to my liking. Fortunately customizability is exactly Emacs’ core philosophy which means I get to do it with freedom. The following are just a few of them which I think might be helpful to others.
Paste rich format text from web pages into Org mode
When I first started to use Org mode there was a glaring deficit to me, which is the inability to paste any formatted text from web pages. If everything pasted becomes plain, much of the original information would be lost, and this would be a great hindrance for anybody wanting to use Org mode exclusively. Luckily this is Emacs where everything is possible. After I posed the question on StackExchange, user @mankoff was quick to come up with a solution using perl and pandoc. Note that this version of code is tested on OS X, but for other systems the difference should only be the utility to call in order to retrieve text from system clipboard, which in this case is osascript.
There are actually a couple of points worthy of further attention. First is that pandoc automatically hard-wraps its output at 70 characters per line. I normally just soft-wrap my notes so I added the option --no-wrap command there.
Then note that pandoc automatically uses Non-breaking space instead of normal space when converting formatted inline text(bold, italics, code etc.), which is not recognized by orgmodeby default. Non-breaking space has to be added to org-emphasis-regexp-components in order for those texts to be formatted correctly in orgmode. The following is my code for that. There seems to be two space characters each line because non-breaking space looks exactly the same as normal space as rendered here, but Emacs recognizes it with a underline.
With this function copy-and-pasting in Org mode supports rich format text as other note-taking apps.
Set non-monospace font for normal notes
Nobody likes to read normal notes in monospace. But the source code snippets better be aligned. Is there a way to achieve both? Definitely. The variable-pitch-mode in Emacs is built just for that.
Remember to first set variable-pitch face in your configuration. Just M-x customize-face on variable-pitch would do. I use Helvetica Neue as I think it’s easiest on the eyes when you stare at it all day. It’s easy to change when you want. The following code was added by customize after I set the font:
Add hook to turn on variable-pitch-mode when the target major modes are enabled. Note I set org-code etc. to be fixed-pitch so that they’re still monospace.
There is a nice post offering a more detailed configuration and claiming to preserve more font attributes such as org table coloring. However for general purposes the simpler configuration should suffice.
Monospace together with normal font:
Hiding Text-Formatting Markers
This is probably more of a matter of personal taste but I don’t really like seeing text-formatting markers all around. Why display /text/ when you can just have text. Turns out there’s a nice option in Org mode: org-hide-emphasis-markers which helps exactly in that. Just M-x customize-variable and then set org-hide-emphasis-markers to t. I didn’t remember seeing this option when I was reading org manual and only stumbled upon it later, so I thought listing it out might help.
Inserting timestamp for each note.
One potentially useful feature of normal note-taking apps is the ability to have the notes timestamped automatically. In Org mode one possible way of doing so is to utilize org-insert-heading-hook. You can define a function insert-inactive-timestamp as
I am fine with having the timestamp in the header directly. So I just have
There’s another solution in an Org mode mail list discussion which inserts the timestamp into the body:
Automatic pull/push for MobileOrg
Traditionally there’s been open source projects called MobileOrg to bring org mode’s functionality to your smartphones. Both Android and iOS versions utilize the same interface to interact with Emacs, which requires manually “pushing” and “pulling” changes from Emacs. There are a few timer-interval based auto-sync solutions out there. However, as I normally don’t use my phone to take notes with my laptop on, just simply pulling on Emacs initialization and pushing on shutting down works surprisingly well,
Note: Recently there has been an Android app called Orgzly which directly syncs your .org files instead of going through the push/pull interface defined for MobileOrg. I find it more refined and actively developed than the previous MobileOrg Android. It’s not open-source, but currently it feels a better app and I’ll be using it for now.
I was used to automatically starting the next pomodoro when using other apps, as you’ll often forget to do it manually when a break finishes. There was an issue in org-pomodoro which prevented this from being configured, but now it’s fixed and you can just use org-pomodoro-break-finished-hook to achieve it:
I’m only starting to have fun with Org mode so I might find some other interesting customizations later and list them here.
(Note: This post is more a kind of rambling personal recount/opinion, so if you’re already an Org mode user and are just interested in some Org mode customizations you might want to see this.)
I’m obsessed with note-taking. I’ve lost count of the number of diary books I wrote as a kid. And back when I was into reading newspapers, I would spend hours each night, carefully cutting and pasting newspaper articles into notebooks, accompanied with my comments; I easily accumulated hundred of pages of paperclips in this fashion. Thus it’s only natural that I immediately set out to find the perfect note-taking app for me after getting into the world of programming. However, the process was unexpectedly fraught with setbacks and angst, until I finally met Emacs Org mode, which is everything a programmer(or possibly any serious note-taker) could ever dream of. I’m amazed by how my daily efficiency massively improved upon using Org mode, how other note-taking apps are just kind of reinventing the wheel, and how I didn’t get to know it earlier. It’s not only for programmers either. I’ve seen many people using Emacs and Org mode for their scientific research, creative writing, personal journal, or whatever purpose out there.
My first note-taking app was OneNote. It was great - it had multiple layers of organization comprising of Notebook, Section, Page, and even Subpage. I gladly wrote a whole lot of notes on books/movies/interesting stuffs I would find on the web. It was an integral part of my life when I first learnt English and prepared for university application. However at that time Microsoft was known to be stubborn to not offer their products outside of Windows. So when I switched to Mac as many programmers would, OneNote was just out of the picture.
Evernote was all the rage at that time. However I didn’t have much fondness for it from the very beginning. It has always seemed to me that Evernote pays more attention to marketing and appealing to “business” users rather than spending serious efforts in the product itself. For a starter, its note organization was amateurish compared with OneNote’s multi-layered approach. It just doesn’t make much sense if all I can do is to throw various notes into one “Notebook” if I were reading several multi-chaptered books, and each chapter has various subtopics. The dealbreaker for me, however, was its incredibly sluggish and unresponsive synchronization. I, being an obsessive-compulsive note-taker, had easily accumulated more than ten thousand notes. Every once in a while Evernote would just automatically initiate its sync progress, freezing the UI in the process and leaving me unable to type anything for 15 seconds. Hell, any half-decent undergrad programmer would tell you that letting a background operation hang the UI is a beginner’s mistake. How could a company with hundreds of millions of dollars in funding not even get such basics straight?
After my unhappy years with Evernote, finally, as a part of its strategy shift, Microsoft began to offer an OS X version of OneNote. I thought it would mark the return of my happy time with note-taking. Wrong. What I didn’t realize was that programming has long been a part of my life. I constantly need to note down fragments of code, reason with them and write down my own observations; I may also take notes with mathematical notations for more theoretical contents (mostly in LaTeX). Besides, I sometimes need to compose and record my own code artifacts, to review and maybe reuse them later. Both Evernote and OneNote were doing a terrible job in that regard. OneNote only offered a simple “Code” formatting, without any syntax highlighting nor language support, and if you paste anything from the Internet, the format was usually broken in one way or another. Evernote didn’t even offer option for you to spontaneously type in code at all, though the pasting support was better. Simply put, it’s hard for me to imagine any programmer-notetaker being happy with the limited support provided.
I set out to search for viable solutions. Markdown-based note-taking software was the first thing that came to my mind. There were a few excellent apps for markdown note-taking, such as Mou, MacDown, and Kobito. They all do a decent job in providing syntax highlighting support and flexible markup. The problem was with structural organization - They were mostly nothing more than single-file editors. Searching, comparing and archiving contents quickly became a nightmare.
Then I came across an app called Quiver, which is actually a decent enough app considering it’s solely a one-man effort. It has an Evernote-esque interface. But what sets it apart is its support for syntax highlighting, Markdown and LaTeX, as it utilizes AceEditor in its core; It even supports Vim/Emacs keybindings. I was happy with it for a while, until I finally got to use Org mode after I picked up Emacs through the excellent Spacemacs project. I was immediately hooked and wondered why I didn’t get to use it sooner:
Its original functionality set already outclasses everything that’s out there: With org-babel you edit source code snippets directly in Emacs major modes for each language seamlessly. If you want, you can even just run the snippet on-the-fly. With org-clock and org-pomodoro coupled with TODO you have a personal project-management + time-tracking suite much better integrated and flexible than the countless (closed-sourced) solutions out there; I no longer have to endure the craziness of using one app to keep track of my tasks, another to log my time, and yet another to take notes related to those tasks. With tools like helm-ack I can search my whole notes repository in a flash with good old ack which is simply more efficient than search functionalities in shiny modern apps. But that’s not even the most important part: No matter how good a program is, there are always things that don’t suit you 100% and on which you want to tweak, which Org mode allows you to do, a lot. This has made me a much happier Org user.
Coming to think about it, I guess one big reason why it had not occurred to me to try out Org mode at all despite it being around for a long time is really that I’ve been a Vim guy. From what I read around the web, I got this notion that Vim and Emacs are just two distinct things that you both try out and then decide to stick with one of them, incompatible with anything that the other camp has to offer. It’s not that I didn’t try out Emacs before, I did and thought the text editing experience was clumsy compared with Vim, and the conceptual overhead of learning another system is huge. However, following this line of thought I just got lazy and didn’t even try out various Vim emulation tools out there such as the excellent evil, not to mention Org mode; this proved to be a huge mistake. To a large part I’m grateful to the open-source project Spacemacs, which offers an Emacs Kit focused on Vim emulation. You can immediately start to use it as if it were Vim, just with awesome Emacs features such as Org mode supported natively (with Vim-like keybindings). Later you’ll naturally feel the thirst to tweak its behaviors here and there, thus gradually becoming proficient with all the quirks and inner mechanism of Emacs. It also has another set of configuration focused on holy-mode for those who don’t use Vim. Whether you’re familiar with Vim or not, I definitely recommend you to check it out as a nice entry point to the wonderful world of Emacs and Org mode.
What a kind of makes me uneasy is the perceived tendency where Vim and Emacs are just simply being heralded less and less by programmers of the new generation, while in fact they’re still unparalleled in many aspects. I know my university is nothing compared with top-quality CS schools, but the fact that barely anybody around me uses Vim and Emacs daily, and that a large portion of students have absolutely zero idea what those things are, still staggers me. Also I had also been a SublimeText user for some time, but still, I realized it doesn’t really compare to either native Vim or Emacs. Vim offers unparalleled editing efficiency and freedom which no emulation I’ve seen so far has been able to 100% bring. As of why Emacs is important, a perhaps inappropriate analogy I’ve come up with is owning a car. Emacs embodies this philosophy where you want full ability to customize your own beloved vehicle. You may power up its engine, change its window, paint it a different color… Anything imaginable, you do it with your own hands instead of having to choose from some limited factory preset or ever resorting to any external hands, which will likely inevitably fall short of your expectations one way or another. You are the mechanic here, and I bet many a mechanic’s ultimate dream would be a fully-powered, wholly customized car of his own. Emacs is just such a car, and its whole design is transparent and open to contributions so if you think you might improve it here and there in its next generation you can just do it. Not to mention it’s also free. The only thing that I currently see might come close is Atom, though with the community and accumulated packages for Emacs I don’t see myself using Atom anytime soon.
I’ve read in quite a few places how “true hackers use Emacs”. I didn’t quite buy into it when I only used Vim. But now I’d say I kind of understand what it means, as I already can’t live without Emacs. Its general esotericism might be one of the hindrances for it and Org mode to attain deserved level of public following proportional to its power, but actually there are already a few nice starter kits out there besides the aforementioned Spacemacs which eases the learning curve. You may find them listed at Awesome Emacs, an “Awesome” style community-driven list of Emacs resources.